Oct 31 13:02:37.387642 ip-10-0-137-77 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 13:02:37.844955 ip-10-0-137-77 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 13:02:37.844955 ip-10-0-137-77 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 13:02:37.844955 ip-10-0-137-77 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 13:02:37.844955 ip-10-0-137-77 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 13:02:37.849092 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849006    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:02:37.849092 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849092    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:02:37.849092 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849096    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849100    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849103    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849105    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849108    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849111    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849114    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849117    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849119    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849122    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849125    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849127    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849130    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849132    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849135    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849139    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849142    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849144    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849147    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849149    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:02:37.849190 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849152    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849154    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849157    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849160    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849162    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849164    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849167    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849170    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849172    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849175    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849178    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849181    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849184    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849187    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849190    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849194    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849198    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849203    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849207    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:02:37.849669 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849210    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849213    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849216    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849219    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849222    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849225    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849228    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849230    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849233    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849235    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849238    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849241    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849244    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849246    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849249    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849251    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849254    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849256    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849259    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:02:37.850116 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849261    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849264    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849266    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849269    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849272    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849274    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849278    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849281    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849284    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849287    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849289    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849304    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849309    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849313    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849317    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849320    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849323    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849326    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849330    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:02:37.850585 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849333    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849336    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849338    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849341    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849344    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849346    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849356    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849358    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849361    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849364    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849366    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849369    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849371    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849374    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849377    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849380    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849390    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849393    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849396    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849399    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:02:37.851062 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849402    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849405    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849408    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849411    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849414    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849416    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.849419    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850202    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850209    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850212    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850215    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850218    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850221    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850224    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850227    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850229    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850232    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850234    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850237    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850239    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:02:37.851573 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850242    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850244    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850247    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850249    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850252    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850254    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850257    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850260    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850263    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850266    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850268    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850271    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850274    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850277    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850280    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850282    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850285    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850287    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850305    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:02:37.852057 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850308    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850311    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850314    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850316    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850319    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850321    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850325    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850327    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850330    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850333    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850335    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850338    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850341    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850344    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850346    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850349    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850351    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850354    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850357    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:02:37.852538 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850359    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850362    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850364    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850367    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850369    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850372    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850374    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850378    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850381    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850383    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850386    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850388    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850391    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850393    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850396    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850398    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850401    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850403    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850406    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:02:37.852983 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850408    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850411    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850414    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850417    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850419    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850421    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850424    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850426    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850429    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850433    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850437    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850440    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850445    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850448    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850451    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850453    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850456    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850458    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850461    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:02:37.853464 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850463    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850466    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850468    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850471    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850474    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850476    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850479    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850489    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850494    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850498    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850501    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850504    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850507    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850509    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850514    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850525    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.850528    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851007    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851011    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:02:37.853916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851015    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851017    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851020    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851022    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851025    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851028    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851033    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851035    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851038    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851041    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851043    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851046    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851049    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851052    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851055    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851058    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851061    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851064    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851067    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851069    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:02:37.854384 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851072    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851075    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851077    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851080    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851083    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851085    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851087    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851090    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851093    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851096    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851098    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851101    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851104    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851106    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851109    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851112    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851114    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851116    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851119    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851122    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:02:37.854856 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851124    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851127    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851130    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851132    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851134    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851137    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851140    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851142    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851145    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851147    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851150    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851152    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851155    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851157    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851160    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851162    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851165    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851167    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851170    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:02:37.855346 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851172    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851175    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851178    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851181    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851183    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851186    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851189    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851191    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851194    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851196    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851199    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851201    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851204    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851206    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851209    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851211    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851214    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851216    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851219    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:02:37.855829 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851221    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851224    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851226    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851229    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851232    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851234    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851236    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851239    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851242    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851244    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851247    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851249    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851251    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851254    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851258    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851261    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851263    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851266    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851269    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:02:37.856286 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851273    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851277    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851280    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851283    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851286    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851303    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.851307    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.851936    2571 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.851945    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.851952    2571 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.851958    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.851962    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.851965    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.851970    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.851974    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.851978    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.851981    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.851985    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.851988    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.851991    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.851994    2571 flags.go:64] FLAG: --cgroup-root=""
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.851997    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 13:02:37.856736 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852000    2571 flags.go:64] FLAG: --client-ca-file=""
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852003    2571 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852006    2571 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852011    2571 flags.go:64] FLAG: --cluster-domain=""
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852014    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852018    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852022    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852025    2571 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852031    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852034    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852037    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852041    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852044    2571 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852047    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852050    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852053    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852056    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852061    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852065    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852068    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852071    2571 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852074    2571 flags.go:64] FLAG: --enable-server="true"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852077    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852083    2571 flags.go:64] FLAG: --event-burst="100"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852086    2571 flags.go:64] FLAG: --event-qps="50"
Oct 31 13:02:37.857355 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852089    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852092    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852095    2571 flags.go:64] FLAG: --eviction-hard=""
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852099    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852102    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852106    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852109    2571 flags.go:64] FLAG: --eviction-soft=""
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852112    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852115    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852118    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852121    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852124    2571 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852128    2571 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852130    2571 flags.go:64] FLAG: --feature-gates=""
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852135    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852138    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852143    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852146    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852149    2571 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852152    2571 flags.go:64] FLAG: --help="false"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852155    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852159    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852161    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 13:02:37.857969 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852165    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852168    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852172    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852175    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852178    2571 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852181    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852183    2571 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852186    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852189    2571 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852192    2571 flags.go:64] FLAG: --kube-reserved=""
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852195    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852198    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852201    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852204    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852207    2571 flags.go:64] FLAG: --lock-file=""
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852210    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852212    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852215    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852221    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852224    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852227    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852230    2571 flags.go:64] FLAG: --logging-format="text"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852233    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852237    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 13:02:37.858562 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852240    2571 flags.go:64] FLAG: --manifest-url=""
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852244    2571 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852249    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852252    2571 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852256    2571 flags.go:64] FLAG: --max-pods="110"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852260    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852263    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852266    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852269    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852272    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852275    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852279    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852286    2571 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852289    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852304    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852308    2571 flags.go:64] FLAG: --pod-cidr=""
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852311    2571 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852314    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852317    2571 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852320    2571 flags.go:64] FLAG: --port="10250"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852323    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852326    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0d4d27fe16577aee8"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852330    2571 flags.go:64] FLAG: --qos-reserved=""
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852333    2571 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852336    2571 flags.go:64] FLAG: --register-node="true"
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852339    2571 flags.go:64] FLAG: --register-with-taints=""
Oct 31 13:02:37.859153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852343    2571 flags.go:64] FLAG: --registry-burst="10"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852346    2571 flags.go:64] FLAG: --registry-qps="5"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852349    2571 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852352    2571 flags.go:64] FLAG: --reserved-memory=""
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852356    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852359    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852362    2571 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852366    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852370    2571 flags.go:64] FLAG: --runonce="false"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852373    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852377    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852380    2571 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852383    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852386    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852389    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852392    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852396    2571 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852399    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852402    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852405    2571 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852408    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852411    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852414    2571 flags.go:64] FLAG: --system-cgroups=""
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852417    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852423    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 13:02:37.859809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852426    2571 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852429    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852433    2571 flags.go:64] FLAG: --tls-min-version=""
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852436    2571 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852439    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852442    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852445    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852448    2571 flags.go:64] FLAG: --v="2"
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852452    2571 flags.go:64] FLAG: --version="false"
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852456    2571 flags.go:64] FLAG: --vmodule=""
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852460    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852464    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852603    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852608    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852613    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852616    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852619    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852622    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852625    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852629    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852632    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852635    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:02:37.860394 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852638    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852641    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852644    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852647    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852649    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852652    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852655    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852657    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852660    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852662    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852665    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852668    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852671    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852673    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852676    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852679    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852681    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852684    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852686    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852689    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:02:37.860898 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852694    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852696    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852699    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852701    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852705    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852708    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852710    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852713    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852716    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852719    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852721    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852724    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852728    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852731    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852734    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852737    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852740    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852743    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852746    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:02:37.861407 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852749    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852751    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852755    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852758    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852760    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852763    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852765    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852768    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852770    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852773    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852775    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852778    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852780    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852787    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852790    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852792    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852795    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852799    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852802    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:02:37.861852 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852805    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852808    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852810    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852814    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852817    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852819    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852822    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852825    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852827    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852829    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852832    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852835    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852837    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852840    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852842    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852845    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852847    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852850    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852852    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:02:37.862367 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852855    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852858    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852861    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852863    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852866    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852868    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852871    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852875    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852877    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852880    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852883    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852887    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852890    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852892    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852895    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852897    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852900    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852903    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:02:37.862835 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.852905    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:02:37.863261 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.852933    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 13:02:37.863261 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853004    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:02:37.863261 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853008    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:02:37.863261 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853012    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:02:37.863261 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853015    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:02:37.863261 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853018    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:02:37.863261 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853021    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:02:37.863261 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853023    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:02:37.863261 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853027    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:02:37.863261 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853031    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:02:37.863261 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853034    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:02:37.863261 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853037    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:02:37.863261 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853039    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:02:37.863261 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853042    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:02:37.863261 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853045    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:02:37.863261 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853047    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853050    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853052    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853055    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853057    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853061    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853064    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853067    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853070    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853073    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853076    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853079    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853081    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853084    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853087    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853089    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853092    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853095    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853097    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853100    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:02:37.863670 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853103    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853105    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853108    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853110    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853113    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853115    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853118    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853120    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853124    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853127    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853130    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853133    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853136    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853138    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853141    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853143    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853146    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853152    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:02:37.864141 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853155    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853157    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853160    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853162    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853166    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853169    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853171    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853174    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853176    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853179    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853181    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853184    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853187    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853189    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853192    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853195    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853197    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853200    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853202    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853205    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:02:37.864609 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853207    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853210    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853212    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853215    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853217    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853220    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853222    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853225    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853227    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853230    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853232    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853236    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853239    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853242    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853244    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853247    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853251    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853261    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853264    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853267    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:02:37.865397 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853269    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853272    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853274    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853277    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853280    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853282    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853285    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853288    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853303    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853306    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853309    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853311    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853314    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.853316    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.859803    2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.859821    2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.859832    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.859837    2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859894    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:02:37.865916 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859900    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859904    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859908    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859911    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859914    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859917    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859920    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859922    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859925    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859928    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859930    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859933    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859935    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859939    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859942    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859946    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859948    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859951    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859954    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:02:37.866390 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859957    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859959    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859962    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859964    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859967    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859971    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859973    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859976    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859978    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859982    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859985    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859988    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859990    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859993    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859995    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.859998    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860001    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860003    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860006    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860008    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:02:37.866845 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860011    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860014    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860016    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860019    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860021    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860024    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860026    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860029    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860032    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860034    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860037    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860040    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860042    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860045    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860047    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860050    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860053    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860055    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860058    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:02:37.867337 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860061    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860063    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860066    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860070    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860072    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860075    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860077    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860080    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860082    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860084    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860087    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860089    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860092    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860094    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860097    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860099    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860102    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860104    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860107    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:02:37.867783 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860110    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860112    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860115    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860117    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860120    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860122    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860125    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860127    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860130    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860132    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860135    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860138    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860141    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860143    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860146    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860149    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860152    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860155    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860157    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860160    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:02:37.868229 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860163    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:02:37.868722 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860165    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:02:37.868722 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860168    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:02:37.868722 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860170    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:02:37.868722 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860173    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:02:37.868722 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860175    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:02:37.868722 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860178    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:02:37.868722 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860180    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:02:37.868722 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.860207    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 13:02:37.868722 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860384    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:02:37.868722 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860390    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:02:37.868722 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860393    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:02:37.868722 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860396    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:02:37.868722 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860401    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:02:37.868722 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860405    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:02:37.868722 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860409    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860412    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860415    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860418    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860421    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860423    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860426    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860428    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860431    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860434    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860437    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860439    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860442    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860444    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860447    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860450    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860453    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860455    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860458    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860460    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:02:37.869083 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860463    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860465    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860468    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860470    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860473    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860475    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860478    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860481    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860483    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860486    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860489    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860492    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860494    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860497    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860499    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860501    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860504    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860506    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860509    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860511    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:02:37.869580 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860515    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860517    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860520    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860522    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860525    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860527    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860530    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860533    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860536    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860538    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860541    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860543    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860546    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860548    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860551    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860553    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860555    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860558    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860561    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860563    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:02:37.870042 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860566    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860568    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860571    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860573    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860576    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860578    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860581    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860583    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860586    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860589    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860591    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860594    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860597    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860599    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860602    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860604    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860607    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860610    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860612    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860615    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:02:37.870611 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860617    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860620    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860622    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860625    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860628    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860630    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860633    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860635    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860637    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860640    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860643    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860647    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860649    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860652    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860655    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860657    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860659    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860662    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860664    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:02:37.871086 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860667    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:02:37.871566 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.860690    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 13:02:37.871566 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860720    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:02:37.871566 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860724    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:02:37.871566 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860727    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:02:37.871566 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860730    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:02:37.871566 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860732    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:02:37.871566 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860735    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:02:37.871566 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860738    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:02:37.871566 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860740    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:02:37.871566 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860743    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:02:37.871566 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860746    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:02:37.871566 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860748    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:02:37.871566 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860751    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:02:37.871566 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860753    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:02:37.871566 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860756    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:02:37.871566 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860759    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860761    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860764    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860766    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860769    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860771    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860774    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860776    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860779    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860781    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860784    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860787    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860789    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860792    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860794    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860797    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860799    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860802    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860804    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860806    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:02:37.871971 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860809    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860812    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860814    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860817    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860819    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860822    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860825    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860827    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860829    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860832    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860834    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860837    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860840    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860843    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860845    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860847    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860850    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860852    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860855    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860857    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:02:37.872451 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860860    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860862    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860864    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860867    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860869    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860872    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860875    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860877    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860880    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860883    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860885    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860888    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860890    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860893    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860895    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860897    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860900    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860903    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860906    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860908    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:02:37.872954 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860911    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860914    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860916    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860919    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860921    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860924    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860926    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860929    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860931    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860934    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860936    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860939    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860941    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860945    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860949    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860951    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860954    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860958    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860961    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:02:37.873430 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860965    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860967    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860970    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860973    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860975    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860978    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860981    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860983    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860987    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860991    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860995    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.860998    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:37.861001    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.861055    2571 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.864367    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.866056    2571 server.go:1014] "Starting client certificate rotation"
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.866157    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 13:02:37.873857 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.866195    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 13:02:37.874286 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.867745    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 13:02:37.874286 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.870777    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 13:02:37.882012 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.881994    2571 log.go:25] "Validated CRI v1 runtime API"
Oct 31 13:02:37.886099 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.886082    2571 log.go:25] "Validated CRI v1 image API"
Oct 31 13:02:37.889451 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.889420    2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 13:02:37.892348 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.892327    2571 fs.go:135] Filesystem UUIDs: map[44866b4a-6aa7-4638-a2cf-3d07c10dbe23:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9acc9f80-b538-4aa1-b3d8-41ba6c48aa50:/dev/nvme0n1p3]
Oct 31 13:02:37.892427 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.892346    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 13:02:37.898843 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.898713    2571 manager.go:217] Machine: {Timestamp:2026-10-31 13:02:37.897165338 +0000 UTC m=+0.370163545 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100770 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aa0ca0e97aa1660ea71ffba8a411a SystemUUID:ec2aa0ca-0e97-aa16-60ea-71ffba8a411a BootID:edd0cc28-4c91-4fbc-a087-a1b0bbf3f59b Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cd:fc:f3:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cd:fc:f3:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:86:a3:cf:81:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 13:02:37.898843 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.898833    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 13:02:37.898995 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.898914    2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 13:02:37.901707 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.901679    2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 13:02:37.901852 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.901707    2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-77.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 13:02:37.901916 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.901861    2571 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 13:02:37.901916 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.901868    2571 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 13:02:37.901916 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.901901    2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 13:02:37.901916 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.901914    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 13:02:37.902776 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.902762    2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 13:02:37.906556 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.906542    2571 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 13:02:37.906623 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.906607    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 13:02:37.906758 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.906746    2571 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 13:02:37.906798 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.906781    2571 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 13:02:37.906798 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.906796    2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 13:02:37.906869 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.906805    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 13:02:37.906869 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.906814    2571 kubelet.go:405] "Adding apiserver pod source"
Oct 31 13:02:37.906869 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.906822    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 13:02:37.907437 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.907422    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 13:02:37.909912 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.909876    2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 13:02:37.920499 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.920479    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 13:02:37.921205 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.921193    2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 13:02:37.921249 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.921224    2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 13:02:37.921398 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.921387    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 13:02:37.921430 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.921403    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 13:02:37.921430 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.921410    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 13:02:37.921430 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.921416    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 13:02:37.921430 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.921422    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 13:02:37.921430 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.921427    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 13:02:37.921942 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.921934    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 13:02:37.921973 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.921943    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 13:02:37.921973 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.921951    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 13:02:37.921973 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.921956    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 13:02:37.921973 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.921965    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 13:02:37.921973 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.921973    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 13:02:37.922541 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.922530    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 13:02:37.922573 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.922542    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 13:02:37.926123 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.926110    2571 server.go:1263] "Started kubelet"
Oct 31 13:02:37.926353 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.926227    2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 13:02:37.926353 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.926187    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 13:02:37.926353 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.926306    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 13:02:37.926969 ip-10-0-137-77 systemd[1]: Started Kubernetes Kubelet.
Oct 31 13:02:37.928695 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.928680    2571 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 13:02:37.930072 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.930054    2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 13:02:37.932433 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:37.932414    2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 13:02:37.933633 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.933616    2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 13:02:37.933633 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.933628    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 13:02:37.933821 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:37.933797    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:37.934365 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.934311    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 13:02:37.934452 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.934331    2571 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 13:02:37.934452 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.934455    2571 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 13:02:37.934576 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.934528    2571 factory.go:55] Registering systemd factory
Oct 31 13:02:37.934576 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.934545    2571 factory.go:223] Registration of the systemd container factory successfully
Oct 31 13:02:37.934689 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.934670    2571 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 13:02:37.934689 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.934685    2571 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 13:02:37.934879 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.934753    2571 factory.go:153] Registering CRI-O factory
Oct 31 13:02:37.934879 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.934764    2571 factory.go:223] Registration of the crio container factory successfully
Oct 31 13:02:37.934879 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.934808    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 13:02:37.934879 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.934822    2571 factory.go:103] Registering Raw factory
Oct 31 13:02:37.934879 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.934833    2571 manager.go:1199] Started watching for new ooms in manager
Oct 31 13:02:37.935709 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.935692    2571 manager.go:319] Starting recovery of all containers
Oct 31 13:02:37.936456 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:37.936412    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 13:02:37.937075 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.937056    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-77.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 13:02:37.937170 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:37.937151    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-77.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 13:02:37.937785 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:37.936497    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-77.us-east-2.compute.internal.18e39edf28fbd8ab  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-77.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-77.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 13:02:37.926078635 +0000 UTC m=+0.399076843,LastTimestamp:2026-10-31 13:02:37.926078635 +0000 UTC m=+0.399076843,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-77.us-east-2.compute.internal,}"
Oct 31 13:02:37.940438 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:37.940408    2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-77.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 13:02:37.940540 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:37.940516    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 13:02:37.946003 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.945983    2571 manager.go:324] Recovery completed
Oct 31 13:02:37.952857 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.952841    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:02:37.954519 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.954503    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:02:37.954611 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.954535    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:02:37.954611 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.954550    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:02:37.955111 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.955094    2571 cpu_manager.go:226] "Starting" policy="none"
Oct 31 13:02:37.955111 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.955102    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:02:37.955111 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.955108    2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 13:02:37.955284 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.955127    2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 13:02:37.955831 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.955813    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:02:37.955913 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.955844    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:02:37.955913 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.955853    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:02:37.956693 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:37.956632    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-77.us-east-2.compute.internal.18e39edf2aadd079  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-77.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-77.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-77.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 13:02:37.954519161 +0000 UTC m=+0.427517375,LastTimestamp:2026-10-31 13:02:37.954519161 +0000 UTC m=+0.427517375,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-77.us-east-2.compute.internal,}"
Oct 31 13:02:37.957418 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.957405    2571 policy_none.go:50] "Start"
Oct 31 13:02:37.957475 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.957420    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 13:02:37.957475 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.957432    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 13:02:37.959514 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.959502    2571 policy_none.go:44] "Start"
Oct 31 13:02:37.969357 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.969331    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f5bql"
Oct 31 13:02:37.969463 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:37.969326    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-77.us-east-2.compute.internal.18e39edf2aae297b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-77.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-77.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-77.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 13:02:37.954541947 +0000 UTC m=+0.427540156,LastTimestamp:2026-10-31 13:02:37.954541947 +0000 UTC m=+0.427540156,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-77.us-east-2.compute.internal,}"
Oct 31 13:02:37.981610 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.981591    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f5bql"
Oct 31 13:02:37.986086 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:37.985973    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-77.us-east-2.compute.internal.18e39edf2aae5f1b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-77.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-77.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-77.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 13:02:37.954555675 +0000 UTC m=+0.427553891,LastTimestamp:2026-10-31 13:02:37.954555675 +0000 UTC m=+0.427553891,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-77.us-east-2.compute.internal,}"
Oct 31 13:02:37.996909 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:37.996876    2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 13:02:38.000996 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.000982    2571 manager.go:341] "Starting Device Plugin manager"
Oct 31 13:02:38.001581 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:38.001565    2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 13:02:38.001581 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.001578    2571 server.go:87] "Starting device plugin registration server"
Oct 31 13:02:38.001828 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.001815    2571 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 13:02:38.001867 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.001832    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 13:02:38.001966 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.001938    2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 13:02:38.002023 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.002009    2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 13:02:38.002023 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.002018    2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 13:02:38.002634 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:38.002609    2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 13:02:38.002716 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:38.002656    2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:38.062316 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.062158    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 13:02:38.064252 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.063539    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 13:02:38.064252 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.063558    2571 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 13:02:38.064252 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.063580    2571 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 13:02:38.064252 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:38.063620    2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 13:02:38.071047 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.071029    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:02:38.102195 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.102128    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:02:38.103127 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.103102    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:02:38.103218 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.103136    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:02:38.103218 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.103166    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:02:38.103218 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.103204    2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.111304 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.111280    2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.111370 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:38.111315    2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-77.us-east-2.compute.internal\": node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:38.123602 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:38.123579    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:38.164239 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.164207    2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal"]
Oct 31 13:02:38.164418 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.164316    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:02:38.165323 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.165309    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:02:38.165407 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.165343    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:02:38.165407 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.165358    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:02:38.167569 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.167553    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:02:38.167707 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.167690    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.167752 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.167724    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:02:38.168281 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.168258    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:02:38.168281 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.168258    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:02:38.168469 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.168289    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:02:38.168469 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.168320    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:02:38.168469 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.168326    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:02:38.168469 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.168344    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:02:38.170384 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.170368    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.170444 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.170403    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:02:38.171143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.171127    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:02:38.171205 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.171167    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:02:38.171205 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.171183    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:02:38.199114 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:38.199091    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found" node="ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.201643 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:38.201628    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found" node="ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.223930 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:38.223910    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:38.236195 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.236176    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e08a66f57394b6ed04cdde42605f9314-config\") pod \"kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"e08a66f57394b6ed04cdde42605f9314\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.236276 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.236203    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71ed2bb6e8396f1c7d68918710e8a4ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"71ed2bb6e8396f1c7d68918710e8a4ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.236276 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.236221    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71ed2bb6e8396f1c7d68918710e8a4ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"71ed2bb6e8396f1c7d68918710e8a4ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.324791 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:38.324752    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:38.337108 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.337087    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71ed2bb6e8396f1c7d68918710e8a4ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"71ed2bb6e8396f1c7d68918710e8a4ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.337181 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.337047    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71ed2bb6e8396f1c7d68918710e8a4ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"71ed2bb6e8396f1c7d68918710e8a4ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.337181 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.337152    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71ed2bb6e8396f1c7d68918710e8a4ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"71ed2bb6e8396f1c7d68918710e8a4ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.337181 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.337171    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e08a66f57394b6ed04cdde42605f9314-config\") pod \"kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"e08a66f57394b6ed04cdde42605f9314\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.337285 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.337196    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e08a66f57394b6ed04cdde42605f9314-config\") pod \"kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"e08a66f57394b6ed04cdde42605f9314\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.337285 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.337223    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71ed2bb6e8396f1c7d68918710e8a4ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"71ed2bb6e8396f1c7d68918710e8a4ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.425515 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:38.425486    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:38.502005 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.501975    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.505045 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.505025    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:38.526541 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:38.526518    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:38.629371 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:38.629346    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:38.730158 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:38.730095    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:38.836860 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:38.836832    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:38.858978 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.858957    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:02:38.866081 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.866067    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 13:02:38.866199 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.866181    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 13:02:38.866254 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.866229    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 13:02:38.937072 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:38.937049    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:38.989710 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.989640    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:57:38 +0000 UTC" deadline="2028-06-14 18:22:56.388796181 +0000 UTC"
Oct 31 13:02:38.989710 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:38.989676    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14213h20m17.399130403s"
Oct 31 13:02:39.037531 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:39.037505    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:39.138738 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:39.138708    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:39.154291 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.154265    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:02:39.155707 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.155684    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:02:39.155840 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.155722    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:02:39.155840 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.155734    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:02:39.229028 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:39.228984    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ed2bb6e8396f1c7d68918710e8a4ed.slice/crio-ccb24845f500d1d06d2b8d8d16f767b93a14d4481610bd3fb5ffd9302912c6b4 WatchSource:0}: Error finding container ccb24845f500d1d06d2b8d8d16f767b93a14d4481610bd3fb5ffd9302912c6b4: Status 404 returned error can't find the container with id ccb24845f500d1d06d2b8d8d16f767b93a14d4481610bd3fb5ffd9302912c6b4
Oct 31 13:02:39.233437 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:39.233412    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08a66f57394b6ed04cdde42605f9314.slice/crio-33997268971ea96102f3de9388fad7d021e710012861e1d37576d6f712a8a5d1 WatchSource:0}: Error finding container 33997268971ea96102f3de9388fad7d021e710012861e1d37576d6f712a8a5d1: Status 404 returned error can't find the container with id 33997268971ea96102f3de9388fad7d021e710012861e1d37576d6f712a8a5d1
Oct 31 13:02:39.235516 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.235500    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:02:39.238998 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:39.238981    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:39.339984 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:39.339902    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:39.440175 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:39.440143    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:39.502102 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.502067    2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:02:39.540270 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:39.540234    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:39.641032 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:39.640996    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:39.741783 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:39.741751    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:39.842563 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:39.842525    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:02:39.875623 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.875584    2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:02:39.908724 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.908641    2571 apiserver.go:52] "Watching apiserver"
Oct 31 13:02:39.918662 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.918635    2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 13:02:39.922140 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.919505    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb","openshift-network-operator/iptables-alerter-2d5zt","openshift-cluster-node-tuning-operator/tuned-dfnfj","openshift-image-registry/node-ca-lgcv8","openshift-multus/network-metrics-daemon-pphvg","kube-system/konnectivity-agent-mttz6","openshift-dns/node-resolver-gpl2l","openshift-multus/multus-dskj5","kube-system/global-pull-secret-syncer-hrj9c","openshift-multus/multus-additional-cni-plugins-x27jz","openshift-network-diagnostics/network-check-target-tkc46","openshift-ovn-kubernetes/ovnkube-node-7nv8r"]
Oct 31 13:02:39.924018 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.923992    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:39.926625 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.926602    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:02:39.928716 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.928692    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 13:02:39.928716 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.928712    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 13:02:39.928886 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.928849    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 13:02:39.928953 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.928916    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:39.929504 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.929473    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tmkp5\""
Oct 31 13:02:39.930743 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.930572    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkxw2\""
Oct 31 13:02:39.930743 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.930587    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:02:39.930743 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.930688    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 13:02:39.930743 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.930735    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 13:02:39.931213 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.931191    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:02:39.932500 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.932474    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:02:39.932745 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.932726    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s8mtw\""
Oct 31 13:02:39.933526 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.933223    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 13:02:39.933738 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.933631    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:39.933921 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.933899    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:39.937084 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:39.934011    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pphvg" podUID="cf350772-aff8-4051-820a-dd114e01e441"
Oct 31 13:02:39.937190 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.935672    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 13:02:39.937444 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.935931    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:02:39.937722 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.935985    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 13:02:39.937807 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.936065    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 13:02:39.937807 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.936109    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 13:02:39.937807 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.936190    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lxhpt\""
Oct 31 13:02:39.938437 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.938418    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:02:39.940950 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.940927    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dskj5"
Oct 31 13:02:39.942364 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.942342    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-h72qs\""
Oct 31 13:02:39.943445 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.943426    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:39.943540 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:39.943487    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:02:39.947260 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.947237    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 13:02:39.947359 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.947351    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:02:39.948074 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948054    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:39.948701 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948684    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc5cb602-ea83-47e3-8966-b832277cb06c-agent-certs\") pod \"konnectivity-agent-mttz6\" (UID: \"dc5cb602-ea83-47e3-8966-b832277cb06c\") " pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:02:39.948797 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948708    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0b71943-a5a0-4227-9b85-161248021630-host-slash\") pod \"iptables-alerter-2d5zt\" (UID: \"c0b71943-a5a0-4227-9b85-161248021630\") " pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:02:39.948797 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948723    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysconfig\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:39.948797 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948737    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41b93592-66ac-4e0e-acf9-11271af65e89-serviceca\") pod \"node-ca-lgcv8\" (UID: \"41b93592-66ac-4e0e-acf9-11271af65e89\") " pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:02:39.948797 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948759    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:39.948797 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948776    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-registration-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:39.948797 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948797    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-kubernetes\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:39.949031 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948818    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-host\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:39.949031 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948838    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41b93592-66ac-4e0e-acf9-11271af65e89-host\") pod \"node-ca-lgcv8\" (UID: \"41b93592-66ac-4e0e-acf9-11271af65e89\") " pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:02:39.949031 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948853    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwkhj\" (UniqueName: \"kubernetes.io/projected/c0b71943-a5a0-4227-9b85-161248021630-kube-api-access-qwkhj\") pod \"iptables-alerter-2d5zt\" (UID: \"c0b71943-a5a0-4227-9b85-161248021630\") " pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:02:39.949031 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948867    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc4vh\" (UniqueName: \"kubernetes.io/projected/41b93592-66ac-4e0e-acf9-11271af65e89-kube-api-access-nc4vh\") pod \"node-ca-lgcv8\" (UID: \"41b93592-66ac-4e0e-acf9-11271af65e89\") " pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:02:39.949031 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948891    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnpmh\" (UniqueName: \"kubernetes.io/projected/cf350772-aff8-4051-820a-dd114e01e441-kube-api-access-tnpmh\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:39.949031 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948906    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc5cb602-ea83-47e3-8966-b832277cb06c-konnectivity-ca\") pod \"konnectivity-agent-mttz6\" (UID: \"dc5cb602-ea83-47e3-8966-b832277cb06c\") " pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:02:39.949031 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948926    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-etc-selinux\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:39.949031 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948946    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf67n\" (UniqueName: \"kubernetes.io/projected/1984f406-7241-4b44-99c2-75fcef391d20-kube-api-access-hf67n\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:39.949031 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948965    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-modprobe-d\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:39.949031 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948981    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-lib-modules\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:39.949031 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.948997    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-tmp\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:39.949031 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.949012    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55pqf\" (UniqueName: \"kubernetes.io/projected/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-kube-api-access-55pqf\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:39.949031 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.949028    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-device-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:39.949437 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.949043    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c0b71943-a5a0-4227-9b85-161248021630-iptables-alerter-script\") pod \"iptables-alerter-2d5zt\" (UID: \"c0b71943-a5a0-4227-9b85-161248021630\") " pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:02:39.949437 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.949061    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysctl-d\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:39.949437 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.949086    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysctl-conf\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:39.949437 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.949101    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-systemd\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:39.949437 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.949115    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-run\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:39.949437 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.949139    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-var-lib-kubelet\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:39.949437 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.949153    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-sys-fs\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:39.949437 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.949166    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-sys\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:39.949437 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.949180    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:39.949437 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.949203    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-socket-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:39.949437 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.949217    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-tuned\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:39.951724 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.951693    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:39.951808 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:39.951767    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkc46" podUID="209e1f84-8138-42c3-bd7a-5d1bba5457d0"
Oct 31 13:02:39.954678 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.954322    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"]
Oct 31 13:02:39.954678 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.954486    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:39.958600 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.957961    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 13:02:39.958600 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.957995    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal"]
Oct 31 13:02:39.997035 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.997004    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 13:02:39.997149 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:39.997078    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r72z7"
Oct 31 13:02:40.012980 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.012960    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r72z7"
Oct 31 13:02:40.035925 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.035901    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 13:02:40.038261 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.037969    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 13:02:40.038261 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.038065    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 13:02:40.038261 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.038144    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 13:02:40.038261 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.038199    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-s9d6g\""
Oct 31 13:02:40.038261 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.038218    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 13:02:40.038261 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.038227    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 13:02:40.038261 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.038223    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-glfvs\""
Oct 31 13:02:40.038261 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.038230    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 13:02:40.038261 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.038223    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 13:02:40.038662 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.038260    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 13:02:40.039545 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.039525    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 13:02:40.039640 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.039588    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 13:02:40.039640 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.039596    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r2wwf\""
Oct 31 13:02:40.039640 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.039633    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 13:02:40.039794 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.039649    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 13:02:40.039794 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.039779    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 13:02:40.039919 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.039793    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 13:02:40.039919 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.039851    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 13:02:40.040105 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.039932    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-65v8h\""
Oct 31 13:02:40.040174 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.040156    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 13:02:40.050475 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.050447    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-modprobe-d\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.050580 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.050495    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-kubelet\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.050580 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.050532    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.050580 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.050562    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-device-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:40.050753 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.050589    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c0b71943-a5a0-4227-9b85-161248021630-iptables-alerter-script\") pod \"iptables-alerter-2d5zt\" (UID: \"c0b71943-a5a0-4227-9b85-161248021630\") " pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:02:40.050753 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.050616    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysctl-conf\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.050753 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.050639    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-run\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.050753 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.050662    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-modprobe-d\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.050753 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.050666    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-var-lib-kubelet\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.050753 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.050738    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91204e03-5b1b-4084-a699-ac7b59db4be5-dbus\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:40.050961 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.050784    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-var-lib-kubelet\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.050961 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.050885    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-run\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.050961 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.050919    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-device-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:40.051052 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051007    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysctl-conf\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.051052 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051041    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.051114 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051087    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.051114 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051108    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-sys-fs\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:40.051178 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051142    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-hostroot\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.051178 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051165    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cni-binary-copy\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.051239 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051181    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-systemd\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.051239 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051198    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-ovnkube-config\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.051239 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051208    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c0b71943-a5a0-4227-9b85-161248021630-iptables-alerter-script\") pod \"iptables-alerter-2d5zt\" (UID: \"c0b71943-a5a0-4227-9b85-161248021630\") " pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:02:40.051239 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051235    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-socket-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:40.051361 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051252    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7231ca35-6551-41ea-b341-46f8836425c1-hosts-file\") pod \"node-resolver-gpl2l\" (UID: \"7231ca35-6551-41ea-b341-46f8836425c1\") " pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:02:40.051361 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051268    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-var-lib-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.051361 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051282    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-sys-fs\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:40.051361 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051283    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-log-socket\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.051361 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051313    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc5cb602-ea83-47e3-8966-b832277cb06c-agent-certs\") pod \"konnectivity-agent-mttz6\" (UID: \"dc5cb602-ea83-47e3-8966-b832277cb06c\") " pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:02:40.051361 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051331    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41b93592-66ac-4e0e-acf9-11271af65e89-serviceca\") pod \"node-ca-lgcv8\" (UID: \"41b93592-66ac-4e0e-acf9-11271af65e89\") " pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:02:40.051361 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051356    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-cnibin\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.051599 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051396    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-multus-certs\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.051599 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051420    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-socket-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:40.051599 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051449    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-kubelet\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.051599 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051466    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-slash\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.051599 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051481    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-cni-netd\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.051599 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051501    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:40.051599 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051539    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-kubernetes\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.051599 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051563    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41b93592-66ac-4e0e-acf9-11271af65e89-host\") pod \"node-ca-lgcv8\" (UID: \"41b93592-66ac-4e0e-acf9-11271af65e89\") " pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:02:40.051599 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051583    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-system-cni-dir\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.051599 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051599    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-node-log\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.051943 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051615    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v649w\" (UniqueName: \"kubernetes.io/projected/58af76a6-3167-4627-bb9f-2c172665a8eb-kube-api-access-v649w\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.051943 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051633    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-etc-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.051943 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051649    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.051943 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051670    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/58af76a6-3167-4627-bb9f-2c172665a8eb-ovn-node-metrics-cert\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.051943 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051740    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-kubernetes\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.051943 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051732    2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 13:02:40.051943 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051766    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41b93592-66ac-4e0e-acf9-11271af65e89-host\") pod \"node-ca-lgcv8\" (UID: \"41b93592-66ac-4e0e-acf9-11271af65e89\") " pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:02:40.051943 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:40.051779    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:40.051943 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.051797    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc5cb602-ea83-47e3-8966-b832277cb06c-konnectivity-ca\") pod \"konnectivity-agent-mttz6\" (UID: \"dc5cb602-ea83-47e3-8966-b832277cb06c\") " pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:02:40.051943 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:40.051865    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs podName:cf350772-aff8-4051-820a-dd114e01e441 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:40.55183935 +0000 UTC m=+3.024837560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs") pod "network-metrics-daemon-pphvg" (UID: "cf350772-aff8-4051-820a-dd114e01e441") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:40.052391 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052025    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-etc-selinux\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:40.052391 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052079    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf67n\" (UniqueName: \"kubernetes.io/projected/1984f406-7241-4b44-99c2-75fcef391d20-kube-api-access-hf67n\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:40.052391 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052110    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-system-cni-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.052391 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052180    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-cni-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.052391 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052128    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41b93592-66ac-4e0e-acf9-11271af65e89-serviceca\") pod \"node-ca-lgcv8\" (UID: \"41b93592-66ac-4e0e-acf9-11271af65e89\") " pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:02:40.052391 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052223    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-k8s-cni-cncf-io\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.052391 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052249    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-etc-selinux\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:40.052391 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052267    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7wbz\" (UniqueName: \"kubernetes.io/projected/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-kube-api-access-z7wbz\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.052391 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052346    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-lib-modules\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.052784 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052436    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-tmp\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.052784 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052477    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55pqf\" (UniqueName: \"kubernetes.io/projected/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-kube-api-access-55pqf\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.052784 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052528    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc5cb602-ea83-47e3-8966-b832277cb06c-konnectivity-ca\") pod \"konnectivity-agent-mttz6\" (UID: \"dc5cb602-ea83-47e3-8966-b832277cb06c\") " pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:02:40.052784 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052572    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7231ca35-6551-41ea-b341-46f8836425c1-tmp-dir\") pod \"node-resolver-gpl2l\" (UID: \"7231ca35-6551-41ea-b341-46f8836425c1\") " pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:02:40.052784 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052619    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-systemd-units\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.052784 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052622    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-lib-modules\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.052784 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052652    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-run-netns\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.052784 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052689    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-env-overrides\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.052784 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052742    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysctl-d\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.052784 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052795    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-systemd\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052836    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv7sk\" (UniqueName: \"kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk\") pod \"network-check-target-tkc46\" (UID: \"209e1f84-8138-42c3-bd7a-5d1bba5457d0\") " pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052844    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-systemd\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052872    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysctl-d\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052887    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052912    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-sys\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052945    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk2rw\" (UniqueName: \"kubernetes.io/projected/7231ca35-6551-41ea-b341-46f8836425c1-kube-api-access-qk2rw\") pod \"node-resolver-gpl2l\" (UID: \"7231ca35-6551-41ea-b341-46f8836425c1\") " pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052963    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-netns\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.052987    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-cni-multus\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053025    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053038    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-sys\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053059    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8zz6\" (UniqueName: \"kubernetes.io/projected/52a6b59d-1f07-4fc3-9974-2da3a2d37676-kube-api-access-r8zz6\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053097    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053163    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-tuned\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053195    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-cni-binary-copy\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053231    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-daemon-config\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053257    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0b71943-a5a0-4227-9b85-161248021630-host-slash\") pod \"iptables-alerter-2d5zt\" (UID: \"c0b71943-a5a0-4227-9b85-161248021630\") " pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:02:40.053307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053309    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysconfig\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053341    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0b71943-a5a0-4227-9b85-161248021630-host-slash\") pod \"iptables-alerter-2d5zt\" (UID: \"c0b71943-a5a0-4227-9b85-161248021630\") " pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053379    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cnibin\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053410    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-registration-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053446    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysconfig\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053459    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-host\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053487    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-registration-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053521    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-cni-bin\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053543    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-conf-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053560    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-etc-kubernetes\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053671    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91204e03-5b1b-4084-a699-ac7b59db4be5-kubelet-config\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053677    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-host\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053693    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-ovn\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053733    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-cni-bin\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053745    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053780    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwkhj\" (UniqueName: \"kubernetes.io/projected/c0b71943-a5a0-4227-9b85-161248021630-kube-api-access-qwkhj\") pod \"iptables-alerter-2d5zt\" (UID: \"c0b71943-a5a0-4227-9b85-161248021630\") " pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053826    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc4vh\" (UniqueName: \"kubernetes.io/projected/41b93592-66ac-4e0e-acf9-11271af65e89-kube-api-access-nc4vh\") pod \"node-ca-lgcv8\" (UID: \"41b93592-66ac-4e0e-acf9-11271af65e89\") " pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:02:40.054143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.053894    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-socket-dir-parent\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.054938 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.054011    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.054938 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.054056    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnpmh\" (UniqueName: \"kubernetes.io/projected/cf350772-aff8-4051-820a-dd114e01e441-kube-api-access-tnpmh\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:40.054938 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.054093    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-ovnkube-script-lib\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.056456 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.056436    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc5cb602-ea83-47e3-8966-b832277cb06c-agent-certs\") pod \"konnectivity-agent-mttz6\" (UID: \"dc5cb602-ea83-47e3-8966-b832277cb06c\") " pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:02:40.056860 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.056831    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-tmp\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.057409 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.057367    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-tuned\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.068848 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.068819    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf67n\" (UniqueName: \"kubernetes.io/projected/1984f406-7241-4b44-99c2-75fcef391d20-kube-api-access-hf67n\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:40.070268 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.070185    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc4vh\" (UniqueName: \"kubernetes.io/projected/41b93592-66ac-4e0e-acf9-11271af65e89-kube-api-access-nc4vh\") pod \"node-ca-lgcv8\" (UID: \"41b93592-66ac-4e0e-acf9-11271af65e89\") " pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:02:40.070951 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.070923    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnpmh\" (UniqueName: \"kubernetes.io/projected/cf350772-aff8-4051-820a-dd114e01e441-kube-api-access-tnpmh\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:40.071064 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.071042    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwkhj\" (UniqueName: \"kubernetes.io/projected/c0b71943-a5a0-4227-9b85-161248021630-kube-api-access-qwkhj\") pod \"iptables-alerter-2d5zt\" (UID: \"c0b71943-a5a0-4227-9b85-161248021630\") " pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:02:40.071427 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.071378    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal" event={"ID":"e08a66f57394b6ed04cdde42605f9314","Type":"ContainerStarted","Data":"33997268971ea96102f3de9388fad7d021e710012861e1d37576d6f712a8a5d1"}
Oct 31 13:02:40.071706 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.071688    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55pqf\" (UniqueName: \"kubernetes.io/projected/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-kube-api-access-55pqf\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.072865 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.072814    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal" event={"ID":"71ed2bb6e8396f1c7d68918710e8a4ed","Type":"ContainerStarted","Data":"ccb24845f500d1d06d2b8d8d16f767b93a14d4481610bd3fb5ffd9302912c6b4"}
Oct 31 13:02:40.154536 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154494    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-ovn\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.154536 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154536    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-cni-bin\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.154772 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154566    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-socket-dir-parent\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.154772 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154591    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.154772 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154603    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-ovn\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.154772 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154616    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-ovnkube-script-lib\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.154772 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154644    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-kubelet\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.154772 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154676    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-kubelet\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.154772 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154683    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.154772 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154706    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-cni-bin\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.154772 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154721    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91204e03-5b1b-4084-a699-ac7b59db4be5-dbus\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:40.154772 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154751    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-socket-dir-parent\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.154772 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154751    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.155237 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154780    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.155237 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154806    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-hostroot\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.155237 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154823    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.155237 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.154834    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cni-binary-copy\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.155237 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155004    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91204e03-5b1b-4084-a699-ac7b59db4be5-dbus\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:40.155237 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155149    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-systemd\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.155536 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155231    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-hostroot\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.155536 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155272    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-ovnkube-config\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.155536 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155334    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7231ca35-6551-41ea-b341-46f8836425c1-hosts-file\") pod \"node-resolver-gpl2l\" (UID: \"7231ca35-6551-41ea-b341-46f8836425c1\") " pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:02:40.155536 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155356    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-var-lib-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.155536 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155372    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-log-socket\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.155536 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155390    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-cnibin\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.155536 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155405    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-multus-certs\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.155536 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155419    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-kubelet\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.155536 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155434    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-slash\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.155536 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155449    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-cni-netd\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.155536 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155487    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-system-cni-dir\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.155536 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155505    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7231ca35-6551-41ea-b341-46f8836425c1-hosts-file\") pod \"node-resolver-gpl2l\" (UID: \"7231ca35-6551-41ea-b341-46f8836425c1\") " pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155548    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-kubelet\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155549    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-multus-certs\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155559    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-node-log\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155512    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-node-log\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155280    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-systemd\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155595    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-system-cni-dir\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155596    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-slash\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155632    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v649w\" (UniqueName: \"kubernetes.io/projected/58af76a6-3167-4627-bb9f-2c172665a8eb-kube-api-access-v649w\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155638    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-cni-netd\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155663    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-etc-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155690    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-cnibin\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155692    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155709    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-var-lib-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155733    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-etc-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155734    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-log-socket\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155747    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/58af76a6-3167-4627-bb9f-2c172665a8eb-ovn-node-metrics-cert\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155764    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155800    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-system-cni-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155833    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-cni-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155861    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-system-cni-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155866    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-k8s-cni-cncf-io\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155901    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7wbz\" (UniqueName: \"kubernetes.io/projected/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-kube-api-access-z7wbz\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155903    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-cni-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.155923    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-k8s-cni-cncf-io\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156069    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7231ca35-6551-41ea-b341-46f8836425c1-tmp-dir\") pod \"node-resolver-gpl2l\" (UID: \"7231ca35-6551-41ea-b341-46f8836425c1\") " pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156098    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-systemd-units\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156122    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-run-netns\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156145    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-env-overrides\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156174    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7sk\" (UniqueName: \"kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk\") pod \"network-check-target-tkc46\" (UID: \"209e1f84-8138-42c3-bd7a-5d1bba5457d0\") " pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156199    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156225    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk2rw\" (UniqueName: \"kubernetes.io/projected/7231ca35-6551-41ea-b341-46f8836425c1-kube-api-access-qk2rw\") pod \"node-resolver-gpl2l\" (UID: \"7231ca35-6551-41ea-b341-46f8836425c1\") " pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156249    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-netns\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156281    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-cni-multus\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.156899 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156300    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156332    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8zz6\" (UniqueName: \"kubernetes.io/projected/52a6b59d-1f07-4fc3-9974-2da3a2d37676-kube-api-access-r8zz6\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156351    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-cni-binary-copy\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156465    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-daemon-config\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156480    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-systemd-units\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156534    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cnibin\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156565    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-cni-bin\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156628    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-conf-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156626    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156655    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-etc-kubernetes\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156681    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91204e03-5b1b-4084-a699-ac7b59db4be5-kubelet-config\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156804    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91204e03-5b1b-4084-a699-ac7b59db4be5-kubelet-config\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156849    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156888    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-ovnkube-script-lib\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.156958    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.157013    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-run-netns\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.157057    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-cni-multus\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.157655 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.157097    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-netns\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.158307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.157096    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.158307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.157145    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-cni-bin\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.158307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.157186    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cnibin\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.158307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.157184    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-conf-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.158307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.157222    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-etc-kubernetes\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.158307 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:40.157273    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:40.158307 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:40.157325    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret podName:91204e03-5b1b-4084-a699-ac7b59db4be5 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:40.657307834 +0000 UTC m=+3.130306043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret") pod "global-pull-secret-syncer-hrj9c" (UID: "91204e03-5b1b-4084-a699-ac7b59db4be5") : object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:40.158307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.157456    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-cni-binary-copy\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.158307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.157610    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-daemon-config\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.158307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.157956    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cni-binary-copy\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.158307 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.158070    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-ovnkube-config\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.158923 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.158703    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-env-overrides\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.158923 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.158774    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7231ca35-6551-41ea-b341-46f8836425c1-tmp-dir\") pod \"node-resolver-gpl2l\" (UID: \"7231ca35-6551-41ea-b341-46f8836425c1\") " pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:02:40.161013 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.160987    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/58af76a6-3167-4627-bb9f-2c172665a8eb-ovn-node-metrics-cert\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.167137 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:40.167112    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 13:02:40.167233 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:40.167142    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 13:02:40.167233 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:40.167156    2571 projected.go:196] Error preparing data for projected volume kube-api-access-dv7sk for pod openshift-network-diagnostics/network-check-target-tkc46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:40.167233 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:40.167219    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk podName:209e1f84-8138-42c3-bd7a-5d1bba5457d0 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:40.667201664 +0000 UTC m=+3.140199882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dv7sk" (UniqueName: "kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk") pod "network-check-target-tkc46" (UID: "209e1f84-8138-42c3-bd7a-5d1bba5457d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:40.169906 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.169884    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7wbz\" (UniqueName: \"kubernetes.io/projected/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-kube-api-access-z7wbz\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.170412 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.170386    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v649w\" (UniqueName: \"kubernetes.io/projected/58af76a6-3167-4627-bb9f-2c172665a8eb-kube-api-access-v649w\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.172439 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.172415    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk2rw\" (UniqueName: \"kubernetes.io/projected/7231ca35-6551-41ea-b341-46f8836425c1-kube-api-access-qk2rw\") pod \"node-resolver-gpl2l\" (UID: \"7231ca35-6551-41ea-b341-46f8836425c1\") " pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:02:40.176076 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.176049    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8zz6\" (UniqueName: \"kubernetes.io/projected/52a6b59d-1f07-4fc3-9974-2da3a2d37676-kube-api-access-r8zz6\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.245865 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.245814    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:02:40.253559 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.253535    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:02:40.257641 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:40.257609    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1984f406_7241_4b44_99c2_75fcef391d20.slice/crio-0ffd7d88d151c8943f6d8be4d3eedd3a685287828150bf4e48d8f6e1c49bc81e WatchSource:0}: Error finding container 0ffd7d88d151c8943f6d8be4d3eedd3a685287828150bf4e48d8f6e1c49bc81e: Status 404 returned error can't find the container with id 0ffd7d88d151c8943f6d8be4d3eedd3a685287828150bf4e48d8f6e1c49bc81e
Oct 31 13:02:40.265583 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.265554    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:02:40.265871 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:40.265841    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b71943_a5a0_4227_9b85_161248021630.slice/crio-9253a55016ae2ece3d5b6b87601d30ec447c842a6b46d6e31ca96ac3c777f7b7 WatchSource:0}: Error finding container 9253a55016ae2ece3d5b6b87601d30ec447c842a6b46d6e31ca96ac3c777f7b7: Status 404 returned error can't find the container with id 9253a55016ae2ece3d5b6b87601d30ec447c842a6b46d6e31ca96ac3c777f7b7
Oct 31 13:02:40.272862 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.272841    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:02:40.276386 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:40.276351    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9a93ef_e1e7_40a7_82ba_51d8a1cabe5e.slice/crio-a6a40863627844525d623e45ccc9239a51d9358afd00d4171011386cfb7c8d91 WatchSource:0}: Error finding container a6a40863627844525d623e45ccc9239a51d9358afd00d4171011386cfb7c8d91: Status 404 returned error can't find the container with id a6a40863627844525d623e45ccc9239a51d9358afd00d4171011386cfb7c8d91
Oct 31 13:02:40.279963 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.279934    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:02:40.283775 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:40.283750    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b93592_66ac_4e0e_acf9_11271af65e89.slice/crio-c80f48133c950dec0e233909b963cd0cc96431128ace1a9e433afccf4e6277b8 WatchSource:0}: Error finding container c80f48133c950dec0e233909b963cd0cc96431128ace1a9e433afccf4e6277b8: Status 404 returned error can't find the container with id c80f48133c950dec0e233909b963cd0cc96431128ace1a9e433afccf4e6277b8
Oct 31 13:02:40.290321 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:40.290289    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5cb602_ea83_47e3_8966_b832277cb06c.slice/crio-5f5198b5f2308e0b0ff3e2a419abe5102d986aebb4cb3e465a7196b59a975aa4 WatchSource:0}: Error finding container 5f5198b5f2308e0b0ff3e2a419abe5102d986aebb4cb3e465a7196b59a975aa4: Status 404 returned error can't find the container with id 5f5198b5f2308e0b0ff3e2a419abe5102d986aebb4cb3e465a7196b59a975aa4
Oct 31 13:02:40.294928 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.294899    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:02:40.303156 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.303139    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dskj5"
Oct 31 13:02:40.305521 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:40.305491    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7231ca35_6551_41ea_b341_46f8836425c1.slice/crio-1b31cae7f3687a6535930f31be851f095451286b328692760cc6fc6091d3753c WatchSource:0}: Error finding container 1b31cae7f3687a6535930f31be851f095451286b328692760cc6fc6091d3753c: Status 404 returned error can't find the container with id 1b31cae7f3687a6535930f31be851f095451286b328692760cc6fc6091d3753c
Oct 31 13:02:40.314282 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:40.314248    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ed0a8a_fd41_44ec_8c25_fa7421d3b360.slice/crio-adcde96d03f7d70b8276d274c287de71b7fa3d0a4f50f3c85000cbd579628aed WatchSource:0}: Error finding container adcde96d03f7d70b8276d274c287de71b7fa3d0a4f50f3c85000cbd579628aed: Status 404 returned error can't find the container with id adcde96d03f7d70b8276d274c287de71b7fa3d0a4f50f3c85000cbd579628aed
Oct 31 13:02:40.332242 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.332214    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:02:40.343740 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:40.343706    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a6b59d_1f07_4fc3_9974_2da3a2d37676.slice/crio-fe6bc3dfc6c7cecac1cc8804b658445b3bdcaa3b17bfe4c626ad61f2ea0e2a7e WatchSource:0}: Error finding container fe6bc3dfc6c7cecac1cc8804b658445b3bdcaa3b17bfe4c626ad61f2ea0e2a7e: Status 404 returned error can't find the container with id fe6bc3dfc6c7cecac1cc8804b658445b3bdcaa3b17bfe4c626ad61f2ea0e2a7e
Oct 31 13:02:40.344832 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.344798    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:02:40.355698 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:02:40.355675    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58af76a6_3167_4627_bb9f_2c172665a8eb.slice/crio-cd46746273f2cb2f5eb0958dd8ca05d009bc6f0619dd403def4fb826f2baaf78 WatchSource:0}: Error finding container cd46746273f2cb2f5eb0958dd8ca05d009bc6f0619dd403def4fb826f2baaf78: Status 404 returned error can't find the container with id cd46746273f2cb2f5eb0958dd8ca05d009bc6f0619dd403def4fb826f2baaf78
Oct 31 13:02:40.382367 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.382334    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:02:40.559729 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.559653    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:40.559892 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:40.559815    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:40.559948 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:40.559893    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs podName:cf350772-aff8-4051-820a-dd114e01e441 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:41.559872524 +0000 UTC m=+4.032870731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs") pod "network-metrics-daemon-pphvg" (UID: "cf350772-aff8-4051-820a-dd114e01e441") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:40.660777 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.660241    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:40.660777 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:40.660388    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:40.660777 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:40.660464    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret podName:91204e03-5b1b-4084-a699-ac7b59db4be5 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:41.660443686 +0000 UTC m=+4.133441884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret") pod "global-pull-secret-syncer-hrj9c" (UID: "91204e03-5b1b-4084-a699-ac7b59db4be5") : object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:40.761499 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:40.761467    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7sk\" (UniqueName: \"kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk\") pod \"network-check-target-tkc46\" (UID: \"209e1f84-8138-42c3-bd7a-5d1bba5457d0\") " pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:40.761680 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:40.761662    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 13:02:40.761761 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:40.761690    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 13:02:40.761761 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:40.761705    2571 projected.go:196] Error preparing data for projected volume kube-api-access-dv7sk for pod openshift-network-diagnostics/network-check-target-tkc46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:40.761866 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:40.761762    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk podName:209e1f84-8138-42c3-bd7a-5d1bba5457d0 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:41.76174333 +0000 UTC m=+4.234741542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dv7sk" (UniqueName: "kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk") pod "network-check-target-tkc46" (UID: "209e1f84-8138-42c3-bd7a-5d1bba5457d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:41.017211 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:41.017099    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:57:40 +0000 UTC" deadline="2028-07-14 21:40:58.733618812 +0000 UTC"
Oct 31 13:02:41.017211 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:41.017173    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14936h38m17.716456576s"
Oct 31 13:02:41.063862 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:41.063818    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:41.064046 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:41.063968    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:02:41.107472 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:41.107321    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r" event={"ID":"58af76a6-3167-4627-bb9f-2c172665a8eb","Type":"ContainerStarted","Data":"cd46746273f2cb2f5eb0958dd8ca05d009bc6f0619dd403def4fb826f2baaf78"}
Oct 31 13:02:41.117224 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:41.117193    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dskj5" event={"ID":"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360","Type":"ContainerStarted","Data":"adcde96d03f7d70b8276d274c287de71b7fa3d0a4f50f3c85000cbd579628aed"}
Oct 31 13:02:41.121033 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:41.120966    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mttz6" event={"ID":"dc5cb602-ea83-47e3-8966-b832277cb06c","Type":"ContainerStarted","Data":"5f5198b5f2308e0b0ff3e2a419abe5102d986aebb4cb3e465a7196b59a975aa4"}
Oct 31 13:02:41.133813 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:41.133779    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lgcv8" event={"ID":"41b93592-66ac-4e0e-acf9-11271af65e89","Type":"ContainerStarted","Data":"c80f48133c950dec0e233909b963cd0cc96431128ace1a9e433afccf4e6277b8"}
Oct 31 13:02:41.135489 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:41.135460    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dfnfj" event={"ID":"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e","Type":"ContainerStarted","Data":"a6a40863627844525d623e45ccc9239a51d9358afd00d4171011386cfb7c8d91"}
Oct 31 13:02:41.138050 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:41.137975    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2d5zt" event={"ID":"c0b71943-a5a0-4227-9b85-161248021630","Type":"ContainerStarted","Data":"9253a55016ae2ece3d5b6b87601d30ec447c842a6b46d6e31ca96ac3c777f7b7"}
Oct 31 13:02:41.143269 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:41.143209    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb" event={"ID":"1984f406-7241-4b44-99c2-75fcef391d20","Type":"ContainerStarted","Data":"0ffd7d88d151c8943f6d8be4d3eedd3a685287828150bf4e48d8f6e1c49bc81e"}
Oct 31 13:02:41.148175 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:41.148143    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x27jz" event={"ID":"52a6b59d-1f07-4fc3-9974-2da3a2d37676","Type":"ContainerStarted","Data":"fe6bc3dfc6c7cecac1cc8804b658445b3bdcaa3b17bfe4c626ad61f2ea0e2a7e"}
Oct 31 13:02:41.151859 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:41.151811    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gpl2l" event={"ID":"7231ca35-6551-41ea-b341-46f8836425c1","Type":"ContainerStarted","Data":"1b31cae7f3687a6535930f31be851f095451286b328692760cc6fc6091d3753c"}
Oct 31 13:02:41.558682 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:41.558408    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:02:41.572027 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:41.571338    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:41.572027 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:41.571539    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:41.572027 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:41.571602    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs podName:cf350772-aff8-4051-820a-dd114e01e441 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:43.571583509 +0000 UTC m=+6.044581767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs") pod "network-metrics-daemon-pphvg" (UID: "cf350772-aff8-4051-820a-dd114e01e441") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:41.672448 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:41.672406    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:41.672644 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:41.672607    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:41.672711 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:41.672677    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret podName:91204e03-5b1b-4084-a699-ac7b59db4be5 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:43.672657427 +0000 UTC m=+6.145655626 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret") pod "global-pull-secret-syncer-hrj9c" (UID: "91204e03-5b1b-4084-a699-ac7b59db4be5") : object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:41.773702 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:41.773663    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7sk\" (UniqueName: \"kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk\") pod \"network-check-target-tkc46\" (UID: \"209e1f84-8138-42c3-bd7a-5d1bba5457d0\") " pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:41.774001 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:41.773876    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 13:02:41.774001 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:41.773897    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 13:02:41.774001 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:41.773912    2571 projected.go:196] Error preparing data for projected volume kube-api-access-dv7sk for pod openshift-network-diagnostics/network-check-target-tkc46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:41.774001 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:41.773976    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk podName:209e1f84-8138-42c3-bd7a-5d1bba5457d0 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:43.773956282 +0000 UTC m=+6.246954485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dv7sk" (UniqueName: "kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk") pod "network-check-target-tkc46" (UID: "209e1f84-8138-42c3-bd7a-5d1bba5457d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:42.017500 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:42.017386    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:57:40 +0000 UTC" deadline="2028-04-02 17:18:27.249174434 +0000 UTC"
Oct 31 13:02:42.017500 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:42.017453    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12460h15m45.231725843s"
Oct 31 13:02:42.065602 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:42.065542    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:42.065779 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:42.065632    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:42.065779 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:42.065735    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkc46" podUID="209e1f84-8138-42c3-bd7a-5d1bba5457d0"
Oct 31 13:02:42.066034 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:42.065993    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pphvg" podUID="cf350772-aff8-4051-820a-dd114e01e441"
Oct 31 13:02:42.181030 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:42.180292    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal" event={"ID":"e08a66f57394b6ed04cdde42605f9314","Type":"ContainerStarted","Data":"2d4eff280414325dbddceb438588253f4c644b0c9baf4a084d0e1f9e2f88471b"}
Oct 31 13:02:42.189326 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:42.188606    2571 generic.go:356] "Generic (PLEG): container finished" podID="71ed2bb6e8396f1c7d68918710e8a4ed" containerID="436dce848fbda3bf0615f40c0b2e9191dd87cf2a0018481296db4b516b85c770" exitCode=0
Oct 31 13:02:42.189326 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:42.188655    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal" event={"ID":"71ed2bb6e8396f1c7d68918710e8a4ed","Type":"ContainerDied","Data":"436dce848fbda3bf0615f40c0b2e9191dd87cf2a0018481296db4b516b85c770"}
Oct 31 13:02:42.219812 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:42.219738    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal" podStartSLOduration=2.219716408 podStartE2EDuration="2.219716408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:42.200133838 +0000 UTC m=+4.673132063" watchObservedRunningTime="2026-10-31 13:02:42.219716408 +0000 UTC m=+4.692714630"
Oct 31 13:02:43.064653 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:43.064612    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:43.065116 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:43.064746    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:02:43.214442 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:43.214362    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal" event={"ID":"71ed2bb6e8396f1c7d68918710e8a4ed","Type":"ContainerStarted","Data":"ed3857d1c329f17abb2b228f69ec835e938ff347c855aab46901756b2d80bee0"}
Oct 31 13:02:43.588841 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:43.588794    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:43.591651 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:43.591608    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:43.591798 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:43.591693    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs podName:cf350772-aff8-4051-820a-dd114e01e441 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:47.591673641 +0000 UTC m=+10.064671853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs") pod "network-metrics-daemon-pphvg" (UID: "cf350772-aff8-4051-820a-dd114e01e441") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:43.690144 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:43.690110    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:43.690332 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:43.690231    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:43.690395 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:43.690387    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret podName:91204e03-5b1b-4084-a699-ac7b59db4be5 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:47.690277613 +0000 UTC m=+10.163275816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret") pod "global-pull-secret-syncer-hrj9c" (UID: "91204e03-5b1b-4084-a699-ac7b59db4be5") : object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:43.790972 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:43.790932    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7sk\" (UniqueName: \"kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk\") pod \"network-check-target-tkc46\" (UID: \"209e1f84-8138-42c3-bd7a-5d1bba5457d0\") " pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:43.791216 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:43.791149    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 13:02:43.791216 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:43.791171    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 13:02:43.791216 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:43.791184    2571 projected.go:196] Error preparing data for projected volume kube-api-access-dv7sk for pod openshift-network-diagnostics/network-check-target-tkc46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:43.791419 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:43.791239    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk podName:209e1f84-8138-42c3-bd7a-5d1bba5457d0 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:47.791222318 +0000 UTC m=+10.264220563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dv7sk" (UniqueName: "kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk") pod "network-check-target-tkc46" (UID: "209e1f84-8138-42c3-bd7a-5d1bba5457d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:44.065554 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:44.065475    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:44.066015 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:44.065473    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:44.066015 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:44.065621    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pphvg" podUID="cf350772-aff8-4051-820a-dd114e01e441"
Oct 31 13:02:44.066015 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:44.065675    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkc46" podUID="209e1f84-8138-42c3-bd7a-5d1bba5457d0"
Oct 31 13:02:44.224695 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:44.223342    2571 generic.go:356] "Generic (PLEG): container finished" podID="52a6b59d-1f07-4fc3-9974-2da3a2d37676" containerID="f6842a9b919f5b76312a37ac743212119f326f9c7a0ab3e8acdb9b8f2b9a77ba" exitCode=0
Oct 31 13:02:44.224695 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:44.224429    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x27jz" event={"ID":"52a6b59d-1f07-4fc3-9974-2da3a2d37676","Type":"ContainerDied","Data":"f6842a9b919f5b76312a37ac743212119f326f9c7a0ab3e8acdb9b8f2b9a77ba"}
Oct 31 13:02:44.253663 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:44.253593    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal" podStartSLOduration=4.253572539 podStartE2EDuration="4.253572539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:43.235499233 +0000 UTC m=+5.708497451" watchObservedRunningTime="2026-10-31 13:02:44.253572539 +0000 UTC m=+6.726570757"
Oct 31 13:02:45.064058 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:45.064019    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:45.064243 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:45.064145    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:02:45.231679 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:45.231634    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mttz6" event={"ID":"dc5cb602-ea83-47e3-8966-b832277cb06c","Type":"ContainerStarted","Data":"7da3eda24e219c91ac70fb7850d00efbd7c30cd7c08b4f8831f6c90baf0a17ee"}
Oct 31 13:02:45.251452 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:45.251132    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mttz6" podStartSLOduration=3.388896024 podStartE2EDuration="7.251113103s" totalImagesPullingTime="3.862217079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:45.250484971 +0000 UTC m=+7.723483203" watchObservedRunningTime="2026-10-31 13:02:45.251113103 +0000 UTC m=+7.724111321"
Oct 31 13:02:46.067687 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:46.066992    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:46.067687 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:46.067125    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkc46" podUID="209e1f84-8138-42c3-bd7a-5d1bba5457d0"
Oct 31 13:02:46.067687 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:46.067509    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:46.067687 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:46.067625    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pphvg" podUID="cf350772-aff8-4051-820a-dd114e01e441"
Oct 31 13:02:46.236157 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:46.236116    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lgcv8" event={"ID":"41b93592-66ac-4e0e-acf9-11271af65e89","Type":"ContainerStarted","Data":"21213415b1729b87d1816811d32cf548b929bbbddd2b486489e41e75288c6396"}
Oct 31 13:02:46.254196 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:46.254151    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lgcv8" podStartSLOduration=3.441225285 podStartE2EDuration="8.254136847s" totalImagesPullingTime="4.812911562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:46.253469814 +0000 UTC m=+8.726468032" watchObservedRunningTime="2026-10-31 13:02:46.254136847 +0000 UTC m=+8.727135087"
Oct 31 13:02:47.063884 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:47.063852    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:47.064066 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:47.063992    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:02:47.247576 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:47.247031    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb" event={"ID":"1984f406-7241-4b44-99c2-75fcef391d20","Type":"ContainerStarted","Data":"b0ad5ac721126a1f88fdf2268d053c15089296a8eb216269d2a4267b80e612f3"}
Oct 31 13:02:47.623756 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:47.623712    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:47.623944 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:47.623886    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:47.623993 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:47.623964    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs podName:cf350772-aff8-4051-820a-dd114e01e441 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:55.623942962 +0000 UTC m=+18.096941163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs") pod "network-metrics-daemon-pphvg" (UID: "cf350772-aff8-4051-820a-dd114e01e441") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:47.725304 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:47.724812    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:47.725304 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:47.724965    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:47.725304 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:47.725018    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret podName:91204e03-5b1b-4084-a699-ac7b59db4be5 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:55.725005422 +0000 UTC m=+18.198003630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret") pod "global-pull-secret-syncer-hrj9c" (UID: "91204e03-5b1b-4084-a699-ac7b59db4be5") : object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:47.808739 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:47.808703    2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 13:02:47.825782 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:47.825740    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7sk\" (UniqueName: \"kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk\") pod \"network-check-target-tkc46\" (UID: \"209e1f84-8138-42c3-bd7a-5d1bba5457d0\") " pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:47.825960 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:47.825904    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 13:02:47.825960 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:47.825920    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 13:02:47.825960 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:47.825934    2571 projected.go:196] Error preparing data for projected volume kube-api-access-dv7sk for pod openshift-network-diagnostics/network-check-target-tkc46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:47.826097 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:47.825986    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk podName:209e1f84-8138-42c3-bd7a-5d1bba5457d0 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:55.82597281 +0000 UTC m=+18.298971005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dv7sk" (UniqueName: "kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk") pod "network-check-target-tkc46" (UID: "209e1f84-8138-42c3-bd7a-5d1bba5457d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:48.006311 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:48.006159    2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T13:02:47.80871989Z","UUID":"dc3a7143-1390-4ed0-be2f-3fdd4deea725","Handler":null,"Name":"","Endpoint":""}
Oct 31 13:02:48.007796 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:48.007769    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 13:02:48.007928 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:48.007805    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 13:02:48.020960 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:48.020922    2571 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/systemd-update-utmp-runlevel.service\": RecentStats: unable to find data in memory cache]"
Oct 31 13:02:48.064856 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:48.064828    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:48.064965 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:48.064930    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkc46" podUID="209e1f84-8138-42c3-bd7a-5d1bba5457d0"
Oct 31 13:02:48.065019 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:48.064975    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:48.065091 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:48.065072    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pphvg" podUID="cf350772-aff8-4051-820a-dd114e01e441"
Oct 31 13:02:48.251270 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:48.251221    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dskj5" event={"ID":"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360","Type":"ContainerStarted","Data":"f2a31ab033c93c024d526092488a5e968fe87aaffd2368ea88523af67b021dbb"}
Oct 31 13:02:48.253683 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:48.253650    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb" event={"ID":"1984f406-7241-4b44-99c2-75fcef391d20","Type":"ContainerStarted","Data":"6035098bfc5a5a55ba4bf4892d0b554418ab0bd45e6ddff6a42e9121e32e75c2"}
Oct 31 13:02:49.064907 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:49.064713    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:49.065046 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:49.064998    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:02:49.258344 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:49.258306    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb" event={"ID":"1984f406-7241-4b44-99c2-75fcef391d20","Type":"ContainerStarted","Data":"22a87ce50509dcb9044d99e2eb8b3bc4c8cbe6ef3e5f9316ef97049eb295ec18"}
Oct 31 13:02:49.260179 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:49.260153    2571 generic.go:356] "Generic (PLEG): container finished" podID="52a6b59d-1f07-4fc3-9974-2da3a2d37676" containerID="17815dfdacf0a85a55813284ba62dd50b172b863c097aecb06ddfe025cf39e54" exitCode=0
Oct 31 13:02:49.260312 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:49.260207    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x27jz" event={"ID":"52a6b59d-1f07-4fc3-9974-2da3a2d37676","Type":"ContainerDied","Data":"17815dfdacf0a85a55813284ba62dd50b172b863c097aecb06ddfe025cf39e54"}
Oct 31 13:02:49.264149 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:49.264121    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gpl2l" event={"ID":"7231ca35-6551-41ea-b341-46f8836425c1","Type":"ContainerStarted","Data":"d74a78e954c9a4ca7f2e74ba5a131b09336f6ac380ba40bb396c73a4b6d62d21"}
Oct 31 13:02:49.283692 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:49.283626    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-dskj5" podStartSLOduration=3.555855996 podStartE2EDuration="11.283609391s" totalImagesPullingTime="7.727753395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:48.272265438 +0000 UTC m=+10.745263656" watchObservedRunningTime="2026-10-31 13:02:49.283609391 +0000 UTC m=+11.756607611"
Oct 31 13:02:49.284256 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:49.284221    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb" podStartSLOduration=3.10712666 podStartE2EDuration="11.284210422s" totalImagesPullingTime="8.177083762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:49.283115097 +0000 UTC m=+11.756113316" watchObservedRunningTime="2026-10-31 13:02:49.284210422 +0000 UTC m=+11.757208643"
Oct 31 13:02:49.701190 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:49.700688    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:02:49.701415 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:49.701246    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:02:49.724111 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:49.724062    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gpl2l" podStartSLOduration=3.841141073 podStartE2EDuration="11.724046196s" totalImagesPullingTime="7.882905123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:49.327116925 +0000 UTC m=+11.800115170" watchObservedRunningTime="2026-10-31 13:02:49.724046196 +0000 UTC m=+12.197044414"
Oct 31 13:02:50.064697 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:50.064666    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:50.064843 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:50.064814    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkc46" podUID="209e1f84-8138-42c3-bd7a-5d1bba5457d0"
Oct 31 13:02:50.064893 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:50.064848    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:50.064965 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:50.064946    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pphvg" podUID="cf350772-aff8-4051-820a-dd114e01e441"
Oct 31 13:02:50.268563 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:50.268484    2571 generic.go:356] "Generic (PLEG): container finished" podID="52a6b59d-1f07-4fc3-9974-2da3a2d37676" containerID="c43e8b1db395c800016096817723f4ff51fc4c45c58b7f8a127e5b19e5c7f839" exitCode=0
Oct 31 13:02:50.269178 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:50.268577    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x27jz" event={"ID":"52a6b59d-1f07-4fc3-9974-2da3a2d37676","Type":"ContainerDied","Data":"c43e8b1db395c800016096817723f4ff51fc4c45c58b7f8a127e5b19e5c7f839"}
Oct 31 13:02:50.270179 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:50.270152    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dfnfj" event={"ID":"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e","Type":"ContainerStarted","Data":"c34d07dd647a3395da62129c5693c673bdfe83304b580fedf417c90c76cea18b"}
Oct 31 13:02:50.271542 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:50.271523    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2d5zt" event={"ID":"c0b71943-a5a0-4227-9b85-161248021630","Type":"ContainerStarted","Data":"78c412f6d6a9b4f15dba71a35cc11edc26bbcce2e62d3068f6a73005814bb9a2"}
Oct 31 13:02:50.314943 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:50.314375    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2d5zt" podStartSLOduration=4.146000242 podStartE2EDuration="12.314356994s" totalImagesPullingTime="8.168356752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:50.313372669 +0000 UTC m=+12.786370887" watchObservedRunningTime="2026-10-31 13:02:50.314356994 +0000 UTC m=+12.787355211"
Oct 31 13:02:50.340217 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:50.340166    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dfnfj" podStartSLOduration=2.721982665 podStartE2EDuration="12.340152467s" totalImagesPullingTime="9.618169802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:50.339861417 +0000 UTC m=+12.812859633" watchObservedRunningTime="2026-10-31 13:02:50.340152467 +0000 UTC m=+12.813150682"
Oct 31 13:02:51.063999 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:51.063969    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:51.064136 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:51.064064    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:02:51.276470 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:51.276431    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x27jz" event={"ID":"52a6b59d-1f07-4fc3-9974-2da3a2d37676","Type":"ContainerStarted","Data":"d994739fee67995b5042e1413a6c04c7df5934e5296e3d0cdbf56e1322c2d3cd"}
Oct 31 13:02:51.557660 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:51.557623    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:02:51.557836 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:51.557752    2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 13:02:51.558275 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:51.558252    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:02:52.064569 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:52.064535    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:52.064746 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:52.064661    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkc46" podUID="209e1f84-8138-42c3-bd7a-5d1bba5457d0"
Oct 31 13:02:52.064746 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:52.064708    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:52.064872 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:52.064827    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pphvg" podUID="cf350772-aff8-4051-820a-dd114e01e441"
Oct 31 13:02:52.281815 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:52.281782    2571 generic.go:356] "Generic (PLEG): container finished" podID="52a6b59d-1f07-4fc3-9974-2da3a2d37676" containerID="d994739fee67995b5042e1413a6c04c7df5934e5296e3d0cdbf56e1322c2d3cd" exitCode=0
Oct 31 13:02:52.282329 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:52.281856    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x27jz" event={"ID":"52a6b59d-1f07-4fc3-9974-2da3a2d37676","Type":"ContainerDied","Data":"d994739fee67995b5042e1413a6c04c7df5934e5296e3d0cdbf56e1322c2d3cd"}
Oct 31 13:02:53.064421 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:53.064206    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:53.064603 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:53.064508    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:02:54.064832 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:54.064791    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:54.064832 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:54.064821    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:54.065410 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:54.064926    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pphvg" podUID="cf350772-aff8-4051-820a-dd114e01e441"
Oct 31 13:02:54.065410 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:54.065039    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkc46" podUID="209e1f84-8138-42c3-bd7a-5d1bba5457d0"
Oct 31 13:02:54.291829 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:54.291794    2571 generic.go:356] "Generic (PLEG): container finished" podID="52a6b59d-1f07-4fc3-9974-2da3a2d37676" containerID="16cfe8274ab2227ebab4171f6859458891420030a6f73c5a822e940d75ed13b9" exitCode=0
Oct 31 13:02:54.292010 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:54.291838    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x27jz" event={"ID":"52a6b59d-1f07-4fc3-9974-2da3a2d37676","Type":"ContainerDied","Data":"16cfe8274ab2227ebab4171f6859458891420030a6f73c5a822e940d75ed13b9"}
Oct 31 13:02:55.064284 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:55.064249    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:55.064471 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:55.064398    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:02:55.297672 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:55.297636    2571 generic.go:356] "Generic (PLEG): container finished" podID="52a6b59d-1f07-4fc3-9974-2da3a2d37676" containerID="ae6467ec5cd1cdf06b4d3531b0e0927e98daa86012876cf0b623bd1020c701f0" exitCode=0
Oct 31 13:02:55.298070 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:55.297685    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x27jz" event={"ID":"52a6b59d-1f07-4fc3-9974-2da3a2d37676","Type":"ContainerDied","Data":"ae6467ec5cd1cdf06b4d3531b0e0927e98daa86012876cf0b623bd1020c701f0"}
Oct 31 13:02:55.690224 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:55.690181    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:55.690419 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:55.690365    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:55.690471 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:55.690430    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs podName:cf350772-aff8-4051-820a-dd114e01e441 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:11.690411167 +0000 UTC m=+34.163409382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs") pod "network-metrics-daemon-pphvg" (UID: "cf350772-aff8-4051-820a-dd114e01e441") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:55.791490 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:55.791450    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:55.791681 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:55.791572    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:55.791681 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:55.791633    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret podName:91204e03-5b1b-4084-a699-ac7b59db4be5 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:11.791615932 +0000 UTC m=+34.264614150 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret") pod "global-pull-secret-syncer-hrj9c" (UID: "91204e03-5b1b-4084-a699-ac7b59db4be5") : object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:55.892361 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:55.892323    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7sk\" (UniqueName: \"kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk\") pod \"network-check-target-tkc46\" (UID: \"209e1f84-8138-42c3-bd7a-5d1bba5457d0\") " pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:55.892530 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:55.892508    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 13:02:55.892530 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:55.892527    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 13:02:55.892530 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:55.892536    2571 projected.go:196] Error preparing data for projected volume kube-api-access-dv7sk for pod openshift-network-diagnostics/network-check-target-tkc46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:55.892766 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:55.892592    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk podName:209e1f84-8138-42c3-bd7a-5d1bba5457d0 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:11.892575146 +0000 UTC m=+34.365573340 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dv7sk" (UniqueName: "kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk") pod "network-check-target-tkc46" (UID: "209e1f84-8138-42c3-bd7a-5d1bba5457d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:56.064667 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:56.064574    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:56.064667 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:56.064628    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:56.064867 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:56.064723    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pphvg" podUID="cf350772-aff8-4051-820a-dd114e01e441"
Oct 31 13:02:56.064921 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:56.064861    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkc46" podUID="209e1f84-8138-42c3-bd7a-5d1bba5457d0"
Oct 31 13:02:56.303927 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:56.303896    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x27jz" event={"ID":"52a6b59d-1f07-4fc3-9974-2da3a2d37676","Type":"ContainerStarted","Data":"2e039dd5e4d24bedb0624831361e237dfbc8b34dea4e2fd140cf881ec389af6e"}
Oct 31 13:02:56.331524 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:56.331412    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x27jz" podStartSLOduration=7.554005441 podStartE2EDuration="18.33139654s" totalImagesPullingTime="10.777391099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:56.329756488 +0000 UTC m=+18.802754708" watchObservedRunningTime="2026-10-31 13:02:56.33139654 +0000 UTC m=+18.804394801"
Oct 31 13:02:57.064604 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:57.064570    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:57.064785 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:57.064684    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:02:57.307640 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:57.307512    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r" event={"ID":"58af76a6-3167-4627-bb9f-2c172665a8eb","Type":"ContainerStarted","Data":"8c6ddecf868bffeae6e6adcb52a72730279a06ff1bf15ac113d39999306c5100"}
Oct 31 13:02:57.308015 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:57.307709    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r" event={"ID":"58af76a6-3167-4627-bb9f-2c172665a8eb","Type":"ContainerStarted","Data":"b73624c079bb71c96991e7d792d11c8293386177cc16bb26225090a6cf072300"}
Oct 31 13:02:58.035814 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:58.035780    2571 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/systemd-update-utmp-runlevel.service\": RecentStats: unable to find data in memory cache]"
Oct 31 13:02:58.065313 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:58.065279    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:02:58.065449 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:58.065377    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkc46" podUID="209e1f84-8138-42c3-bd7a-5d1bba5457d0"
Oct 31 13:02:58.065449 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:58.065426    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:02:58.065574 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:58.065512    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pphvg" podUID="cf350772-aff8-4051-820a-dd114e01e441"
Oct 31 13:02:58.313675 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:58.313572    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r" event={"ID":"58af76a6-3167-4627-bb9f-2c172665a8eb","Type":"ContainerStarted","Data":"dfe1a24f82a6b999647996481a2119827bacc29c48ece8925945d8f797b5e46d"}
Oct 31 13:02:58.314253 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:58.313719    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r" event={"ID":"58af76a6-3167-4627-bb9f-2c172665a8eb","Type":"ContainerStarted","Data":"dfa8813c2dca797d97196912047398ede41c7965d26e0076e23cd7d3c0b88d3d"}
Oct 31 13:02:58.314253 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:58.313797    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r" event={"ID":"58af76a6-3167-4627-bb9f-2c172665a8eb","Type":"ContainerStarted","Data":"72026701fe025c82df122f8d594a6e80c63d08c883425d006ddf4c96f57e9295"}
Oct 31 13:02:58.314253 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:58.313865    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r" event={"ID":"58af76a6-3167-4627-bb9f-2c172665a8eb","Type":"ContainerStarted","Data":"9b2ca81fa1bc8f75131cb4fb1b9f0549fb82367c72405587837318bd214e40a0"}
Oct 31 13:02:59.064625 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:02:59.064593    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:02:59.064788 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:02:59.064705    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:03:00.064604 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:00.064569    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:00.065111 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:00.064710    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pphvg" podUID="cf350772-aff8-4051-820a-dd114e01e441"
Oct 31 13:03:00.065111 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:00.064754    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:03:00.065111 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:00.064821    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkc46" podUID="209e1f84-8138-42c3-bd7a-5d1bba5457d0"
Oct 31 13:03:00.320285 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:00.320206    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r" event={"ID":"58af76a6-3167-4627-bb9f-2c172665a8eb","Type":"ContainerStarted","Data":"4c8f12fac2210dbd246639e76ca9324855617ed1e7dd1b3a9a9128a0be8816ba"}
Oct 31 13:03:01.064768 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:01.064735    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:01.065149 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:01.064837    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:03:02.064334 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:02.064305    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:03:02.064334 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:02.064323    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:02.064587 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:02.064420    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkc46" podUID="209e1f84-8138-42c3-bd7a-5d1bba5457d0"
Oct 31 13:03:02.064587 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:02.064520    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pphvg" podUID="cf350772-aff8-4051-820a-dd114e01e441"
Oct 31 13:03:02.327524 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:02.327451    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r" event={"ID":"58af76a6-3167-4627-bb9f-2c172665a8eb","Type":"ContainerStarted","Data":"0d82d9b1a06bb69ad56d279575e748651b92277e89f1b977cbb1c314e13f782f"}
Oct 31 13:03:02.328261 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:02.327661    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:02.342459 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:02.342432    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:02.367470 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:02.367426    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r" podStartSLOduration=7.553594493 podStartE2EDuration="24.367413027s" totalImagesPullingTime="16.813818534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:02.365793298 +0000 UTC m=+24.838791505" watchObservedRunningTime="2026-10-31 13:03:02.367413027 +0000 UTC m=+24.840411243"
Oct 31 13:03:03.064535 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:03.064509    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:03.064686 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:03.064608    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:03:03.237199 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:03.237165    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tkc46"]
Oct 31 13:03:03.237399 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:03.237315    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:03:03.237478 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:03.237411    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkc46" podUID="209e1f84-8138-42c3-bd7a-5d1bba5457d0"
Oct 31 13:03:03.239797 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:03.239767    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pphvg"]
Oct 31 13:03:03.239918 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:03.239903    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:03.240031 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:03.240013    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pphvg" podUID="cf350772-aff8-4051-820a-dd114e01e441"
Oct 31 13:03:03.240615 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:03.240592    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrj9c"]
Oct 31 13:03:03.329306 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:03.329202    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:03.329692 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:03.329323    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:03:03.329786 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:03.329766    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:03.329834 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:03.329829    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:03.343923 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:03.343903    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:05.064480 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:05.064430    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:05.064886 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:05.064495    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:03:05.064886 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:05.064592    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pphvg" podUID="cf350772-aff8-4051-820a-dd114e01e441"
Oct 31 13:03:05.064886 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:05.064591    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:05.064886 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:05.064707    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkc46" podUID="209e1f84-8138-42c3-bd7a-5d1bba5457d0"
Oct 31 13:03:05.064886 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:05.064762    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:03:07.064756 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:07.064721    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:07.065202 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:07.064833    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:03:07.065202 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:07.064850    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:03:07.065202 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:07.064863    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:07.065202 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:07.064928    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkc46" podUID="209e1f84-8138-42c3-bd7a-5d1bba5457d0"
Oct 31 13:03:07.065202 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:07.065009    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pphvg" podUID="cf350772-aff8-4051-820a-dd114e01e441"
Oct 31 13:03:08.053881 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:08.053850    2571 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/systemd-update-utmp-runlevel.service\": RecentStats: unable to find data in memory cache]"
Oct 31 13:03:09.064282 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.064250    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:03:09.064282 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.064266    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:09.064742 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.064352    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:09.064742 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:09.064368    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkc46" podUID="209e1f84-8138-42c3-bd7a-5d1bba5457d0"
Oct 31 13:03:09.064742 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:09.064420    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrj9c" podUID="91204e03-5b1b-4084-a699-ac7b59db4be5"
Oct 31 13:03:09.064742 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:09.064506    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pphvg" podUID="cf350772-aff8-4051-820a-dd114e01e441"
Oct 31 13:03:09.805843 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.805786    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeReady"
Oct 31 13:03:09.805975 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.805890    2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 13:03:09.845862 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.845838    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wvdhx"]
Oct 31 13:03:09.871335 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.871305    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sqxqp"]
Oct 31 13:03:09.871486 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.871387    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wvdhx"
Oct 31 13:03:09.889456 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.889429    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 13:03:09.889587 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.889427    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 13:03:09.903907 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.903726    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w7wlc"]
Oct 31 13:03:09.904421 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.904405    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-sqxqp"
Oct 31 13:03:09.917150 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.917131    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dswrj"]
Oct 31 13:03:09.917247 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.917236    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w7wlc"
Oct 31 13:03:09.933000 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.932980    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"]
Oct 31 13:03:09.933104 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.933094    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dswrj"
Oct 31 13:03:09.937017 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.936999    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-44698"]
Oct 31 13:03:09.937152 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.937069    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:09.941605 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.941581    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 13:03:09.941699 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.941630    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zrqm9\""
Oct 31 13:03:09.963885 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.963864    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-cpk9z"]
Oct 31 13:03:09.963979 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.963958    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:09.970029 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.970012    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"]
Oct 31 13:03:09.970142 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.970127    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:09.972687 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.972672    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-848dcf9575-7trg4"]
Oct 31 13:03:09.972776 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.972764    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:09.990187 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.990169    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-swrc7"]
Oct 31 13:03:09.990276 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.990263    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:09.991928 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.991898    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:03:09.992019 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.991938    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 13:03:09.992019 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.991986    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 13:03:09.992019 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.991997    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 13:03:09.992019 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.991991    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 13:03:09.992019 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.991998    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 13:03:09.992312 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.992073    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 13:03:09.992312 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.992131    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 13:03:09.992312 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.992165    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 13:03:09.992312 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.992271    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:03:09.997798 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.997782    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 13:03:09.999885 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.999857    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7cm8\" (UniqueName: \"kubernetes.io/projected/ad027e2c-c49c-4c9b-8253-12acc36a1c9c-kube-api-access-n7cm8\") pod \"kas-connection-checker-685895c7c9-wvdhx\" (UID: \"ad027e2c-c49c-4c9b-8253-12acc36a1c9c\") " pod="kube-system/kas-connection-checker-685895c7c9-wvdhx"
Oct 31 13:03:10.000003 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:09.999930    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dmxw\" (UniqueName: \"kubernetes.io/projected/8c98a759-eb02-4d1b-a9f8-f1cb77e1590a-kube-api-access-5dmxw\") pod \"kas-connection-checker-685895c7c9-sqxqp\" (UID: \"8c98a759-eb02-4d1b-a9f8-f1cb77e1590a\") " pod="kube-system/kas-connection-checker-685895c7c9-sqxqp"
Oct 31 13:03:10.000908 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.000893    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 13:03:10.000996 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.000979    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 13:03:10.001054 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.001040    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 13:03:10.006258 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.006241    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"]
Oct 31 13:03:10.006387 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.006372    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:10.024242 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.024224    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"]
Oct 31 13:03:10.024357 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.024342    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:10.042120 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.042091    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 13:03:10.042120 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.042106    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 13:03:10.042277 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.042135    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 13:03:10.042277 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.042178    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 13:03:10.042277 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.042185    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 13:03:10.042277 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.042205    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ftcbc\""
Oct 31 13:03:10.042277 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.042232    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vq7zb\""
Oct 31 13:03:10.042277 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.042234    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 13:03:10.042581 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.042386    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8dvth\""
Oct 31 13:03:10.042581 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.042391    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 13:03:10.042581 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.042409    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 13:03:10.042581 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.042433    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-np6pg\""
Oct 31 13:03:10.042762 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.042618    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 13:03:10.042762 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.042637    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pfgb9\""
Oct 31 13:03:10.042762 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.042658    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 13:03:10.042762 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.042699    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rw847\""
Oct 31 13:03:10.047099 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.047082    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"]
Oct 31 13:03:10.047202 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.047188    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:10.050860 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.050826    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 13:03:10.051149 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.051136    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rlfbc\""
Oct 31 13:03:10.072603 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.072555    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lcmgp"]
Oct 31 13:03:10.072935 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.072653    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:10.088459 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.088440    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 13:03:10.088550 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.088474    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 13:03:10.088550 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.088529    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 13:03:10.088679 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.088548    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:03:10.088735 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.088682    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-694db45b8d-szb8q"]
Oct 31 13:03:10.088868 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.088850    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lcmgp"
Oct 31 13:03:10.088972 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.088955    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 13:03:10.089037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.088955    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 13:03:10.089037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.088961    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 13:03:10.089037 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.089014    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 13:03:10.089180 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.089137    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 13:03:10.089362 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.089342    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:03:10.089440 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.089400    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 13:03:10.089440 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.089421    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:03:10.092445 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.092429    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 13:03:10.096990 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.096975    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 13:03:10.096990 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.096984    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 13:03:10.100595 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.100576    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:10.100680 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.100603    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:10.100680 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.100623    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4x2q\" (UniqueName: \"kubernetes.io/projected/7810b5de-3907-4a6e-b11e-0c631c148438-kube-api-access-f4x2q\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:10.100680 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.100641    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-image-registry-private-configuration\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.100768 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.100684    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:10.100768 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.100730    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:10.100836 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.100826    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:10.100879 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.100865    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqcgc\" (UniqueName: \"kubernetes.io/projected/bf8d1062-0bb5-46f2-98bf-e4d52689294c-kube-api-access-hqcgc\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:10.100919 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.100907    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjwq4\" (UniqueName: \"kubernetes.io/projected/2a1a0fc2-36f2-4635-97b6-97faed580133-kube-api-access-bjwq4\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.100952 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.100940    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-certificates\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.100987 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.100956    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-stats-auth\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:10.101026 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101003    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:10.101060 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101046    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-installation-pull-secrets\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.101091 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101064    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a1a0fc2-36f2-4635-97b6-97faed580133-snapshots\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.101091 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101080    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-bound-sa-token\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.101153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101100    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-default-certificate\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:10.101180 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101149    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-trusted-ca\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.101180 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101176    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a1a0fc2-36f2-4635-97b6-97faed580133-serving-cert\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.101234 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101216    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7cm8\" (UniqueName: \"kubernetes.io/projected/ad027e2c-c49c-4c9b-8253-12acc36a1c9c-kube-api-access-n7cm8\") pod \"kas-connection-checker-685895c7c9-wvdhx\" (UID: \"ad027e2c-c49c-4c9b-8253-12acc36a1c9c\") " pod="kube-system/kas-connection-checker-685895c7c9-wvdhx"
Oct 31 13:03:10.101375 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101358    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1a0fc2-36f2-4635-97b6-97faed580133-service-ca-bundle\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.101409 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101390    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k967\" (UniqueName: \"kubernetes.io/projected/40395d51-6cc7-44b3-8cad-d4a44489e25f-kube-api-access-9k967\") pod \"kas-connection-checker-685895c7c9-dswrj\" (UID: \"40395d51-6cc7-44b3-8cad-d4a44489e25f\") " pod="kube-system/kas-connection-checker-685895c7c9-dswrj"
Oct 31 13:03:10.101443 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101415    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a1a0fc2-36f2-4635-97b6-97faed580133-tmp\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.101475 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101461    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7620273c-a4f1-4e1f-ba26-453913b2e171-ca-trust-extracted\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.101505 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101482    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzhh\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-kube-api-access-mnzhh\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.101538 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101503    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5pvx\" (UniqueName: \"kubernetes.io/projected/a096aa5d-63cc-41a6-ad78-bc64d2d33e94-kube-api-access-x5pvx\") pod \"volume-data-source-validator-58c9f5f46d-w7wlc\" (UID: \"a096aa5d-63cc-41a6-ad78-bc64d2d33e94\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w7wlc"
Oct 31 13:03:10.101569 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101537    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dmxw\" (UniqueName: \"kubernetes.io/projected/8c98a759-eb02-4d1b-a9f8-f1cb77e1590a-kube-api-access-5dmxw\") pod \"kas-connection-checker-685895c7c9-sqxqp\" (UID: \"8c98a759-eb02-4d1b-a9f8-f1cb77e1590a\") " pod="kube-system/kas-connection-checker-685895c7c9-sqxqp"
Oct 31 13:03:10.101569 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101554    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccsk6\" (UniqueName: \"kubernetes.io/projected/c95a45a3-7757-4843-9161-6cdbf64385cd-kube-api-access-ccsk6\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:10.101625 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101579    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73811006-7011-4339-800d-5f1b8a9bbf88-nginx-conf\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:10.101625 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101603    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.101699 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101688    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1a0fc2-36f2-4635-97b6-97faed580133-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.101756 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.101745    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bf8d1062-0bb5-46f2-98bf-e4d52689294c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:10.109098 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.109085    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.109167 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.109154    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:10.112373 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.112355    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wvdhx"]
Oct 31 13:03:10.112456 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.112386    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dswrj"]
Oct 31 13:03:10.112456 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.112407    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sqxqp"]
Oct 31 13:03:10.112456 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.112428    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w7wlc"]
Oct 31 13:03:10.112456 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.112450    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-swrc7"]
Oct 31 13:03:10.112580 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.112472    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l42ds"]
Oct 31 13:03:10.116280 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.115933    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dmxw\" (UniqueName: \"kubernetes.io/projected/8c98a759-eb02-4d1b-a9f8-f1cb77e1590a-kube-api-access-5dmxw\") pod \"kas-connection-checker-685895c7c9-sqxqp\" (UID: \"8c98a759-eb02-4d1b-a9f8-f1cb77e1590a\") " pod="kube-system/kas-connection-checker-685895c7c9-sqxqp"
Oct 31 13:03:10.116443 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.116418    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7cm8\" (UniqueName: \"kubernetes.io/projected/ad027e2c-c49c-4c9b-8253-12acc36a1c9c-kube-api-access-n7cm8\") pod \"kas-connection-checker-685895c7c9-wvdhx\" (UID: \"ad027e2c-c49c-4c9b-8253-12acc36a1c9c\") " pod="kube-system/kas-connection-checker-685895c7c9-wvdhx"
Oct 31 13:03:10.131072 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.131055    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-cpk9z"]
Oct 31 13:03:10.131166 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.131081    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pztwr"]
Oct 31 13:03:10.131224 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.131176    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l42ds"
Oct 31 13:03:10.142187 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.142167    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wr5k7\""
Oct 31 13:03:10.142272 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.142167    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wdnw7\""
Oct 31 13:03:10.142272 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.142192    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 13:03:10.142272 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.142204    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 13:03:10.142272 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.142218    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 13:03:10.142421 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.142334    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7x7tk\""
Oct 31 13:03:10.145490 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.145476    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"]
Oct 31 13:03:10.145534 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.145503    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"]
Oct 31 13:03:10.145534 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.145523    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-848dcf9575-7trg4"]
Oct 31 13:03:10.145592 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.145548    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lcmgp"]
Oct 31 13:03:10.145592 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.145584    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"]
Oct 31 13:03:10.145647 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.145607    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"]
Oct 31 13:03:10.145647 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.145627    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pztwr"]
Oct 31 13:03:10.145700 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.145650    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l42ds"]
Oct 31 13:03:10.145700 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.145672    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"]
Oct 31 13:03:10.145700 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.145685    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-694db45b8d-szb8q"]
Oct 31 13:03:10.145700 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.145699    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-44698"]
Oct 31 13:03:10.145815 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.145648    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:10.151850 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.151832    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qbkfg\""
Oct 31 13:03:10.189010 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.188989    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 13:03:10.189010 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.189000    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 13:03:10.189120 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.189112    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 13:03:10.202434 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.202412    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-default-certificate\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:10.202515 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.202442    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-trusted-ca\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.202515 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.202459    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a1a0fc2-36f2-4635-97b6-97faed580133-serving-cert\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.202515 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.202477    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-config\") pod \"service-ca-operator-544d749c8c-4hrft\" (UID: \"b3bc5c9b-54fb-477f-8c98-9884b1b2e159\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:10.202515 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.202502    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-image-registry-private-configuration\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.202646 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.202527    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.202646 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.202571    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf2v9\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-kube-api-access-gf2v9\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.202721 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.202685    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s\" (UID: \"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:10.202775 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.202729    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-bound-sa-token\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.202823 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.202781    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s\" (UID: \"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:10.202874 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.202857    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1a0fc2-36f2-4635-97b6-97faed580133-service-ca-bundle\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.202982 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.202888    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j64s\" (UniqueName: \"kubernetes.io/projected/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-kube-api-access-7j64s\") pod \"service-ca-operator-544d749c8c-4hrft\" (UID: \"b3bc5c9b-54fb-477f-8c98-9884b1b2e159\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:10.202982 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.202925    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/298ad120-4782-4b13-985f-5af7fb24d97d-ca-trust-extracted\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.203102 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203001    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-serving-cert\") pod \"service-ca-operator-544d749c8c-4hrft\" (UID: \"b3bc5c9b-54fb-477f-8c98-9884b1b2e159\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:10.203102 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203073    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh68v\" (UniqueName: \"kubernetes.io/projected/49b449ac-b419-4bd1-9896-0941450b820a-kube-api-access-kh68v\") pod \"network-check-source-6f78d59f99-lcmgp\" (UID: \"49b449ac-b419-4bd1-9896-0941450b820a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lcmgp"
Oct 31 13:03:10.203193 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203110    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k967\" (UniqueName: \"kubernetes.io/projected/40395d51-6cc7-44b3-8cad-d4a44489e25f-kube-api-access-9k967\") pod \"kas-connection-checker-685895c7c9-dswrj\" (UID: \"40395d51-6cc7-44b3-8cad-d4a44489e25f\") " pod="kube-system/kas-connection-checker-685895c7c9-dswrj"
Oct 31 13:03:10.203193 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203140    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a1a0fc2-36f2-4635-97b6-97faed580133-tmp\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.203193 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203168    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7620273c-a4f1-4e1f-ba26-453913b2e171-ca-trust-extracted\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.203426 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203194    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzhh\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-kube-api-access-mnzhh\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.203426 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203223    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5pvx\" (UniqueName: \"kubernetes.io/projected/a096aa5d-63cc-41a6-ad78-bc64d2d33e94-kube-api-access-x5pvx\") pod \"volume-data-source-validator-58c9f5f46d-w7wlc\" (UID: \"a096aa5d-63cc-41a6-ad78-bc64d2d33e94\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w7wlc"
Oct 31 13:03:10.203426 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203269    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccsk6\" (UniqueName: \"kubernetes.io/projected/c95a45a3-7757-4843-9161-6cdbf64385cd-kube-api-access-ccsk6\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:10.203426 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203307    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73811006-7011-4339-800d-5f1b8a9bbf88-nginx-conf\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:10.203426 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203331    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.203426 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203357    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1a0fc2-36f2-4635-97b6-97faed580133-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.203426 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203386    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bf8d1062-0bb5-46f2-98bf-e4d52689294c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:10.203426 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203412    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4504b20c-a7d8-49e9-a572-9004fce4af1d-serving-cert\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:10.203926 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203436    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn96p\" (UniqueName: \"kubernetes.io/projected/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-kube-api-access-rn96p\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s\" (UID: \"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:10.203926 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203466    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn9c9\" (UniqueName: \"kubernetes.io/projected/4504b20c-a7d8-49e9-a572-9004fce4af1d-kube-api-access-kn9c9\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:10.203926 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203484    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1a0fc2-36f2-4635-97b6-97faed580133-service-ca-bundle\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.203926 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203516    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:10.203926 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203564    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4504b20c-a7d8-49e9-a572-9004fce4af1d-trusted-ca\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:10.203926 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203601    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-registry-certificates\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.203926 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203630    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-installation-pull-secrets\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.203926 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203696    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:10.204332 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203724    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4x2q\" (UniqueName: \"kubernetes.io/projected/7810b5de-3907-4a6e-b11e-0c631c148438-kube-api-access-f4x2q\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:10.204332 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203744    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a1a0fc2-36f2-4635-97b6-97faed580133-tmp\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.204332 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.204066    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7620273c-a4f1-4e1f-ba26-453913b2e171-ca-trust-extracted\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.204332 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.203818    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:10.204332 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.204107    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76fcd8cbb5-tmzzx: secret "image-registry-tls" not found
Oct 31 13:03:10.204332 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.204105    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-image-registry-private-configuration\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.204332 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.204135    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:10.204332 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.204168    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls podName:7620273c-a4f1-4e1f-ba26-453913b2e171 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:10.704150209 +0000 UTC m=+33.177148426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls") pod "image-registry-76fcd8cbb5-tmzzx" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171") : secret "image-registry-tls" not found
Oct 31 13:03:10.204332 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.204196    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:10.204858 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.203800    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-trusted-ca\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.204858 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.203905    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 13:03:10.204858 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.204423    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:10.704408452 +0000 UTC m=+33.177406676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : secret "router-metrics-certs-default" not found
Oct 31 13:03:10.204858 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.204375    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73811006-7011-4339-800d-5f1b8a9bbf88-nginx-conf\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:10.204858 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.204457    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 13:03:10.204858 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.204511    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 13:03:10.204858 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.204514    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls podName:7810b5de-3907-4a6e-b11e-0c631c148438 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:10.704500008 +0000 UTC m=+33.177498211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7sfns" (UID: "7810b5de-3907-4a6e-b11e-0c631c148438") : secret "samples-operator-tls" not found
Oct 31 13:03:10.204858 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.204589    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1a0fc2-36f2-4635-97b6-97faed580133-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.204858 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.204646    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bf8d1062-0bb5-46f2-98bf-e4d52689294c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:10.204858 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.204709    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:10.204858 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.204764    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert podName:73811006-7011-4339-800d-5f1b8a9bbf88 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:10.704749756 +0000 UTC m=+33.177747966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-swrc7" (UID: "73811006-7011-4339-800d-5f1b8a9bbf88") : secret "networking-console-plugin-cert" not found
Oct 31 13:03:10.204858 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.204801    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:10.704790508 +0000 UTC m=+33.177788702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : configmap references non-existent config key: service-ca.crt
Oct 31 13:03:10.204858 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.204841    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqcgc\" (UniqueName: \"kubernetes.io/projected/bf8d1062-0bb5-46f2-98bf-e4d52689294c-kube-api-access-hqcgc\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:10.204858 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.204851    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:10.205521 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.204875    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjwq4\" (UniqueName: \"kubernetes.io/projected/2a1a0fc2-36f2-4635-97b6-97faed580133-kube-api-access-bjwq4\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.205521 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.204954    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-certificates\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.205521 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.204981    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-stats-auth\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:10.205521 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.205008    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4504b20c-a7d8-49e9-a572-9004fce4af1d-config\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:10.205521 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.205033    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-trusted-ca\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.205521 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.205069    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:10.205521 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.205110    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-installation-pull-secrets\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.205521 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.205145    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a1a0fc2-36f2-4635-97b6-97faed580133-snapshots\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.205521 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.205172    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-bound-sa-token\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.205932 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.205732    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a1a0fc2-36f2-4635-97b6-97faed580133-snapshots\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.205932 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.205833    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:10.205932 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.205895    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls podName:bf8d1062-0bb5-46f2-98bf-e4d52689294c nodeName:}" failed. No retries permitted until 2026-10-31 13:03:10.705880955 +0000 UTC m=+33.178879164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-44698" (UID: "bf8d1062-0bb5-46f2-98bf-e4d52689294c") : secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:10.206100 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.205941    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-certificates\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.206251 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.206220    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wvdhx"
Oct 31 13:03:10.206376 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.206316    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-default-certificate\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:10.206671 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.206653    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a1a0fc2-36f2-4635-97b6-97faed580133-serving-cert\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.207819 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.207789    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-installation-pull-secrets\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.207911 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.207849    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-stats-auth\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:10.208003 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.207986    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-image-registry-private-configuration\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.218973 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.218954    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjwq4\" (UniqueName: \"kubernetes.io/projected/2a1a0fc2-36f2-4635-97b6-97faed580133-kube-api-access-bjwq4\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.219110 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.219089    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccsk6\" (UniqueName: \"kubernetes.io/projected/c95a45a3-7757-4843-9161-6cdbf64385cd-kube-api-access-ccsk6\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:10.219255 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.219232    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k967\" (UniqueName: \"kubernetes.io/projected/40395d51-6cc7-44b3-8cad-d4a44489e25f-kube-api-access-9k967\") pod \"kas-connection-checker-685895c7c9-dswrj\" (UID: \"40395d51-6cc7-44b3-8cad-d4a44489e25f\") " pod="kube-system/kas-connection-checker-685895c7c9-dswrj"
Oct 31 13:03:10.219358 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.219341    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4x2q\" (UniqueName: \"kubernetes.io/projected/7810b5de-3907-4a6e-b11e-0c631c148438-kube-api-access-f4x2q\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:10.219948 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.219932    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqcgc\" (UniqueName: \"kubernetes.io/projected/bf8d1062-0bb5-46f2-98bf-e4d52689294c-kube-api-access-hqcgc\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:10.225428 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.225410    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-sqxqp"
Oct 31 13:03:10.228576 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.228553    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5pvx\" (UniqueName: \"kubernetes.io/projected/a096aa5d-63cc-41a6-ad78-bc64d2d33e94-kube-api-access-x5pvx\") pod \"volume-data-source-validator-58c9f5f46d-w7wlc\" (UID: \"a096aa5d-63cc-41a6-ad78-bc64d2d33e94\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w7wlc"
Oct 31 13:03:10.229153 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.229130    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-bound-sa-token\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.232028 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.231935    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzhh\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-kube-api-access-mnzhh\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.235636 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.235612    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w7wlc"
Oct 31 13:03:10.250809 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.250792    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 13:03:10.259187 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.259159    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dswrj"
Oct 31 13:03:10.300777 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.300729    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:10.307658 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.306532    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4504b20c-a7d8-49e9-a572-9004fce4af1d-config\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:10.307658 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.306577    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-trusted-ca\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.307658 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.306605    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bdce0f75-852b-49b6-9da3-f662bb82bf94-config-volume\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:10.307658 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.306657    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdce0f75-852b-49b6-9da3-f662bb82bf94-tmp-dir\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:10.307658 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.306684    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hbtp\" (UniqueName: \"kubernetes.io/projected/bd718975-d0cb-4a21-8383-c29164ae4f31-kube-api-access-4hbtp\") pod \"ingress-canary-l42ds\" (UID: \"bd718975-d0cb-4a21-8383-c29164ae4f31\") " pod="openshift-ingress-canary/ingress-canary-l42ds"
Oct 31 13:03:10.307658 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.306723    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-config\") pod \"service-ca-operator-544d749c8c-4hrft\" (UID: \"b3bc5c9b-54fb-477f-8c98-9884b1b2e159\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:10.307658 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.306747    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-image-registry-private-configuration\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.307658 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.306771    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.307658 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.306798    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2v9\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-kube-api-access-gf2v9\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.307658 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.306826    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s\" (UID: \"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:10.307658 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.306849    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-bound-sa-token\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.307658 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.306874    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6f7q\" (UniqueName: \"kubernetes.io/projected/bdce0f75-852b-49b6-9da3-f662bb82bf94-kube-api-access-t6f7q\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:10.307658 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.306904    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s\" (UID: \"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:10.307658 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.306943    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j64s\" (UniqueName: \"kubernetes.io/projected/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-kube-api-access-7j64s\") pod \"service-ca-operator-544d749c8c-4hrft\" (UID: \"b3bc5c9b-54fb-477f-8c98-9884b1b2e159\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:10.307658 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.306967    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/298ad120-4782-4b13-985f-5af7fb24d97d-ca-trust-extracted\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.307658 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.306993    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-serving-cert\") pod \"service-ca-operator-544d749c8c-4hrft\" (UID: \"b3bc5c9b-54fb-477f-8c98-9884b1b2e159\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:10.308448 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.307017    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh68v\" (UniqueName: \"kubernetes.io/projected/49b449ac-b419-4bd1-9896-0941450b820a-kube-api-access-kh68v\") pod \"network-check-source-6f78d59f99-lcmgp\" (UID: \"49b449ac-b419-4bd1-9896-0941450b820a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lcmgp"
Oct 31 13:03:10.308448 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.307086    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:10.308448 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.307128    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd718975-d0cb-4a21-8383-c29164ae4f31-cert\") pod \"ingress-canary-l42ds\" (UID: \"bd718975-d0cb-4a21-8383-c29164ae4f31\") " pod="openshift-ingress-canary/ingress-canary-l42ds"
Oct 31 13:03:10.308448 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.307177    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4504b20c-a7d8-49e9-a572-9004fce4af1d-serving-cert\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:10.308448 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.307204    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn96p\" (UniqueName: \"kubernetes.io/projected/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-kube-api-access-rn96p\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s\" (UID: \"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:10.308448 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.307233    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn9c9\" (UniqueName: \"kubernetes.io/projected/4504b20c-a7d8-49e9-a572-9004fce4af1d-kube-api-access-kn9c9\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:10.308448 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.307270    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4504b20c-a7d8-49e9-a572-9004fce4af1d-trusted-ca\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:10.308448 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.307312    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-registry-certificates\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.308448 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.307344    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-installation-pull-secrets\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.312111 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.309603    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s\" (UID: \"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:10.312111 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.309807    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-config\") pod \"service-ca-operator-544d749c8c-4hrft\" (UID: \"b3bc5c9b-54fb-477f-8c98-9884b1b2e159\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:10.312111 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.310687    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4504b20c-a7d8-49e9-a572-9004fce4af1d-config\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:10.312111 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.311722    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-trusted-ca\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.322747 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.315386    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-installation-pull-secrets\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.322747 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.315719    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/298ad120-4782-4b13-985f-5af7fb24d97d-ca-trust-extracted\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.322747 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.317551    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4504b20c-a7d8-49e9-a572-9004fce4af1d-trusted-ca\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:10.322747 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.318971    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s\" (UID: \"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:10.322747 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.319459    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-registry-certificates\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.322747 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.319576    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:10.322747 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.319592    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694db45b8d-szb8q: secret "image-registry-tls" not found
Oct 31 13:03:10.322747 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.319645    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls podName:298ad120-4782-4b13-985f-5af7fb24d97d nodeName:}" failed. No retries permitted until 2026-10-31 13:03:10.819626787 +0000 UTC m=+33.292624994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls") pod "image-registry-694db45b8d-szb8q" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d") : secret "image-registry-tls" not found
Oct 31 13:03:10.322747 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.320230    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-serving-cert\") pod \"service-ca-operator-544d749c8c-4hrft\" (UID: \"b3bc5c9b-54fb-477f-8c98-9884b1b2e159\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:10.323807 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.323779    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-image-registry-private-configuration\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.324143 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.324122    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4504b20c-a7d8-49e9-a572-9004fce4af1d-serving-cert\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:10.334384 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.334333    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh68v\" (UniqueName: \"kubernetes.io/projected/49b449ac-b419-4bd1-9896-0941450b820a-kube-api-access-kh68v\") pod \"network-check-source-6f78d59f99-lcmgp\" (UID: \"49b449ac-b419-4bd1-9896-0941450b820a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lcmgp"
Oct 31 13:03:10.335270 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.335063    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn96p\" (UniqueName: \"kubernetes.io/projected/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-kube-api-access-rn96p\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s\" (UID: \"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:10.335928 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.335807    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-bound-sa-token\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.338237 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.338212    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j64s\" (UniqueName: \"kubernetes.io/projected/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-kube-api-access-7j64s\") pod \"service-ca-operator-544d749c8c-4hrft\" (UID: \"b3bc5c9b-54fb-477f-8c98-9884b1b2e159\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:10.338370 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.338213    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn9c9\" (UniqueName: \"kubernetes.io/projected/4504b20c-a7d8-49e9-a572-9004fce4af1d-kube-api-access-kn9c9\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:10.340014 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.339987    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2v9\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-kube-api-access-gf2v9\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.383690 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.383663    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:10.397630 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.397601    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:10.408681 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.408654    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6f7q\" (UniqueName: \"kubernetes.io/projected/bdce0f75-852b-49b6-9da3-f662bb82bf94-kube-api-access-t6f7q\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:10.408816 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.408747    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:10.408816 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.408787    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd718975-d0cb-4a21-8383-c29164ae4f31-cert\") pod \"ingress-canary-l42ds\" (UID: \"bd718975-d0cb-4a21-8383-c29164ae4f31\") " pod="openshift-ingress-canary/ingress-canary-l42ds"
Oct 31 13:03:10.408914 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.408902    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bdce0f75-852b-49b6-9da3-f662bb82bf94-config-volume\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:10.408958 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.408948    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdce0f75-852b-49b6-9da3-f662bb82bf94-tmp-dir\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:10.409005 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.408977    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hbtp\" (UniqueName: \"kubernetes.io/projected/bd718975-d0cb-4a21-8383-c29164ae4f31-kube-api-access-4hbtp\") pod \"ingress-canary-l42ds\" (UID: \"bd718975-d0cb-4a21-8383-c29164ae4f31\") " pod="openshift-ingress-canary/ingress-canary-l42ds"
Oct 31 13:03:10.409646 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.409384    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 13:03:10.409646 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.409451    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls podName:bdce0f75-852b-49b6-9da3-f662bb82bf94 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:10.909432427 +0000 UTC m=+33.382430639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls") pod "dns-default-pztwr" (UID: "bdce0f75-852b-49b6-9da3-f662bb82bf94") : secret "dns-default-metrics-tls" not found
Oct 31 13:03:10.409646 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.409508    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdce0f75-852b-49b6-9da3-f662bb82bf94-tmp-dir\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:10.409646 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.409602    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bdce0f75-852b-49b6-9da3-f662bb82bf94-config-volume\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:10.411325 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.411287    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd718975-d0cb-4a21-8383-c29164ae4f31-cert\") pod \"ingress-canary-l42ds\" (UID: \"bd718975-d0cb-4a21-8383-c29164ae4f31\") " pod="openshift-ingress-canary/ingress-canary-l42ds"
Oct 31 13:03:10.419841 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.419823    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lcmgp"
Oct 31 13:03:10.422477 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.422447    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6f7q\" (UniqueName: \"kubernetes.io/projected/bdce0f75-852b-49b6-9da3-f662bb82bf94-kube-api-access-t6f7q\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:10.423232 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.423197    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hbtp\" (UniqueName: \"kubernetes.io/projected/bd718975-d0cb-4a21-8383-c29164ae4f31-kube-api-access-4hbtp\") pod \"ingress-canary-l42ds\" (UID: \"bd718975-d0cb-4a21-8383-c29164ae4f31\") " pod="openshift-ingress-canary/ingress-canary-l42ds"
Oct 31 13:03:10.438030 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.437689    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:10.505957 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.505783    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l42ds"
Oct 31 13:03:10.540754 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.540719    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sqxqp"]
Oct 31 13:03:10.544067 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.544013    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wvdhx"]
Oct 31 13:03:10.546998 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:03:10.546970    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c98a759_eb02_4d1b_a9f8_f1cb77e1590a.slice/crio-57e5d116ca0671ac2013d8c117f970d0c73e267b60213885528a3d1552422885 WatchSource:0}: Error finding container 57e5d116ca0671ac2013d8c117f970d0c73e267b60213885528a3d1552422885: Status 404 returned error can't find the container with id 57e5d116ca0671ac2013d8c117f970d0c73e267b60213885528a3d1552422885
Oct 31 13:03:10.548879 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:03:10.548852    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad027e2c_c49c_4c9b_8253_12acc36a1c9c.slice/crio-ae7fa7db046613ffd1b237c73aee825e86027157fec24499164ae7e3b7488b4f WatchSource:0}: Error finding container ae7fa7db046613ffd1b237c73aee825e86027157fec24499164ae7e3b7488b4f: Status 404 returned error can't find the container with id ae7fa7db046613ffd1b237c73aee825e86027157fec24499164ae7e3b7488b4f
Oct 31 13:03:10.600121 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.600079    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w7wlc"]
Oct 31 13:03:10.605353 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.605335    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dswrj"]
Oct 31 13:03:10.619148 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.618974    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-cpk9z"]
Oct 31 13:03:10.622655 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:03:10.622625    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40395d51_6cc7_44b3_8cad_d4a44489e25f.slice/crio-5f02d4ca8a60e4f8e2b6872b04f9a1558e59163e3c22f70e90bf1c15c4969f8c WatchSource:0}: Error finding container 5f02d4ca8a60e4f8e2b6872b04f9a1558e59163e3c22f70e90bf1c15c4969f8c: Status 404 returned error can't find the container with id 5f02d4ca8a60e4f8e2b6872b04f9a1558e59163e3c22f70e90bf1c15c4969f8c
Oct 31 13:03:10.622927 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:03:10.622904    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1a0fc2_36f2_4635_97b6_97faed580133.slice/crio-44b73c2b1801c7d3118e6e3a35ddc76b869c3cf3efd15d23f7fe6daf4cd5cd1f WatchSource:0}: Error finding container 44b73c2b1801c7d3118e6e3a35ddc76b869c3cf3efd15d23f7fe6daf4cd5cd1f: Status 404 returned error can't find the container with id 44b73c2b1801c7d3118e6e3a35ddc76b869c3cf3efd15d23f7fe6daf4cd5cd1f
Oct 31 13:03:10.675711 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.675641    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"]
Oct 31 13:03:10.686424 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:03:10.686384    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3bc5c9b_54fb_477f_8c98_9884b1b2e159.slice/crio-5d5ae92c54631a6c2bc8a6a035a0d9c7c471076a30ae65887af26777dc6a2f94 WatchSource:0}: Error finding container 5d5ae92c54631a6c2bc8a6a035a0d9c7c471076a30ae65887af26777dc6a2f94: Status 404 returned error can't find the container with id 5d5ae92c54631a6c2bc8a6a035a0d9c7c471076a30ae65887af26777dc6a2f94
Oct 31 13:03:10.711637 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.711612    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:10.711735 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.711644    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:10.711735 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.711680    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:10.711854 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.711769    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:10.711854 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.711818    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:11.711789068 +0000 UTC m=+34.184787286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : configmap references non-existent config key: service-ca.crt
Oct 31 13:03:10.711854 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.711840    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 13:03:10.712022 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.711892    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert podName:73811006-7011-4339-800d-5f1b8a9bbf88 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:11.711878619 +0000 UTC m=+34.184876815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-swrc7" (UID: "73811006-7011-4339-800d-5f1b8a9bbf88") : secret "networking-console-plugin-cert" not found
Oct 31 13:03:10.712022 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.711893    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:10.712022 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.711887    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:10.712022 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.711927    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:10.712022 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.711948    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76fcd8cbb5-tmzzx: secret "image-registry-tls" not found
Oct 31 13:03:10.712022 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.711949    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls podName:bf8d1062-0bb5-46f2-98bf-e4d52689294c nodeName:}" failed. No retries permitted until 2026-10-31 13:03:11.711934556 +0000 UTC m=+34.184932782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-44698" (UID: "bf8d1062-0bb5-46f2-98bf-e4d52689294c") : secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:10.712022 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.712001    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 13:03:10.712022 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.712024    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls podName:7620273c-a4f1-4e1f-ba26-453913b2e171 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:11.712009733 +0000 UTC m=+34.185007932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls") pod "image-registry-76fcd8cbb5-tmzzx" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171") : secret "image-registry-tls" not found
Oct 31 13:03:10.712454 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.712056    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:11.712036389 +0000 UTC m=+34.185034590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : secret "router-metrics-certs-default" not found
Oct 31 13:03:10.712454 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.712102    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:10.712454 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.712185    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 13:03:10.712454 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.712242    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls podName:7810b5de-3907-4a6e-b11e-0c631c148438 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:11.712228547 +0000 UTC m=+34.185226743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7sfns" (UID: "7810b5de-3907-4a6e-b11e-0c631c148438") : secret "samples-operator-tls" not found
Oct 31 13:03:10.719780 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.719757    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"]
Oct 31 13:03:10.722425 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:03:10.722405    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4504b20c_a7d8_49e9_a572_9004fce4af1d.slice/crio-d911c55ba548c6c1bd2b4c510c219287e2c4c1402d7a679dc648bfe5e6ffd553 WatchSource:0}: Error finding container d911c55ba548c6c1bd2b4c510c219287e2c4c1402d7a679dc648bfe5e6ffd553: Status 404 returned error can't find the container with id d911c55ba548c6c1bd2b4c510c219287e2c4c1402d7a679dc648bfe5e6ffd553
Oct 31 13:03:10.746354 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.746285    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lcmgp"]
Oct 31 13:03:10.748695 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:03:10.748675    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b449ac_b419_4bd1_9896_0941450b820a.slice/crio-88ca662f63a235a6f55dcd3b3d8dfd5b3de712fee6e60ccbee3a45ed68aecd59 WatchSource:0}: Error finding container 88ca662f63a235a6f55dcd3b3d8dfd5b3de712fee6e60ccbee3a45ed68aecd59: Status 404 returned error can't find the container with id 88ca662f63a235a6f55dcd3b3d8dfd5b3de712fee6e60ccbee3a45ed68aecd59
Oct 31 13:03:10.778117 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.778098    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"]
Oct 31 13:03:10.780719 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:03:10.780697    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b29c79_bfdc_4c46_9b31_c2cbf6c0a94a.slice/crio-4fb8c67ec53005d8de87676cd553dbf750cda2a2224001820bc08b616b3f2301 WatchSource:0}: Error finding container 4fb8c67ec53005d8de87676cd553dbf750cda2a2224001820bc08b616b3f2301: Status 404 returned error can't find the container with id 4fb8c67ec53005d8de87676cd553dbf750cda2a2224001820bc08b616b3f2301
Oct 31 13:03:10.825575 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.825554    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l42ds"]
Oct 31 13:03:10.828437 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:03:10.828398    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd718975_d0cb_4a21_8383_c29164ae4f31.slice/crio-803b51df10bd848ac4f47db7333a15708b97fdf101ae7f66f1e3e2c8bbdc93f9 WatchSource:0}: Error finding container 803b51df10bd848ac4f47db7333a15708b97fdf101ae7f66f1e3e2c8bbdc93f9: Status 404 returned error can't find the container with id 803b51df10bd848ac4f47db7333a15708b97fdf101ae7f66f1e3e2c8bbdc93f9
Oct 31 13:03:10.917229 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.914372    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:10.917229 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:10.914480    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:10.917229 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.914772    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 13:03:10.917229 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.914861    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls podName:bdce0f75-852b-49b6-9da3-f662bb82bf94 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:11.914822186 +0000 UTC m=+34.387820391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls") pod "dns-default-pztwr" (UID: "bdce0f75-852b-49b6-9da3-f662bb82bf94") : secret "dns-default-metrics-tls" not found
Oct 31 13:03:10.917229 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.914775    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:10.917229 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.915350    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694db45b8d-szb8q: secret "image-registry-tls" not found
Oct 31 13:03:10.917229 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:10.915438    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls podName:298ad120-4782-4b13-985f-5af7fb24d97d nodeName:}" failed. No retries permitted until 2026-10-31 13:03:11.915420794 +0000 UTC m=+34.388418989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls") pod "image-registry-694db45b8d-szb8q" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d") : secret "image-registry-tls" not found
Oct 31 13:03:11.064342 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.064286    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:03:11.064491 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.064474    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:11.069123 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.069098    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j8xvf\""
Oct 31 13:03:11.069237 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.069168    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5rqjw\""
Oct 31 13:03:11.069237 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.069171    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 13:03:11.351087 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.350992    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l42ds" event={"ID":"bd718975-d0cb-4a21-8383-c29164ae4f31","Type":"ContainerStarted","Data":"803b51df10bd848ac4f47db7333a15708b97fdf101ae7f66f1e3e2c8bbdc93f9"}
Oct 31 13:03:11.352740 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.352716    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-cpk9z" event={"ID":"2a1a0fc2-36f2-4635-97b6-97faed580133","Type":"ContainerStarted","Data":"44b73c2b1801c7d3118e6e3a35ddc76b869c3cf3efd15d23f7fe6daf4cd5cd1f"}
Oct 31 13:03:11.354885 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.354553    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-sqxqp" event={"ID":"8c98a759-eb02-4d1b-a9f8-f1cb77e1590a","Type":"ContainerStarted","Data":"1d5788416b40f5a8219deff13472ca3cea94aac965645e04e373b75c2b6599a8"}
Oct 31 13:03:11.354885 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.354591    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-sqxqp" event={"ID":"8c98a759-eb02-4d1b-a9f8-f1cb77e1590a","Type":"ContainerStarted","Data":"57e5d116ca0671ac2013d8c117f970d0c73e267b60213885528a3d1552422885"}
Oct 31 13:03:11.356432 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.356408    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wvdhx" event={"ID":"ad027e2c-c49c-4c9b-8253-12acc36a1c9c","Type":"ContainerStarted","Data":"7d17c25ac842726c410ec9a0b1180a8cc2d324f06915ba1e6d3536a9139e73c6"}
Oct 31 13:03:11.356528 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.356452    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wvdhx" event={"ID":"ad027e2c-c49c-4c9b-8253-12acc36a1c9c","Type":"ContainerStarted","Data":"ae7fa7db046613ffd1b237c73aee825e86027157fec24499164ae7e3b7488b4f"}
Oct 31 13:03:11.358246 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.358188    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s" event={"ID":"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a","Type":"ContainerStarted","Data":"4fb8c67ec53005d8de87676cd553dbf750cda2a2224001820bc08b616b3f2301"}
Oct 31 13:03:11.360790 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.360769    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft" event={"ID":"b3bc5c9b-54fb-477f-8c98-9884b1b2e159","Type":"ContainerStarted","Data":"5d5ae92c54631a6c2bc8a6a035a0d9c7c471076a30ae65887af26777dc6a2f94"}
Oct 31 13:03:11.363228 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.362762    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dswrj" event={"ID":"40395d51-6cc7-44b3-8cad-d4a44489e25f","Type":"ContainerStarted","Data":"59d32e82b036e6972fec645fc89c965cde8a50b40d1b9d4b7936ccb43c0b0e38"}
Oct 31 13:03:11.363228 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.362803    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dswrj" event={"ID":"40395d51-6cc7-44b3-8cad-d4a44489e25f","Type":"ContainerStarted","Data":"5f02d4ca8a60e4f8e2b6872b04f9a1558e59163e3c22f70e90bf1c15c4969f8c"}
Oct 31 13:03:11.366118 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.366061    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lcmgp" event={"ID":"49b449ac-b419-4bd1-9896-0941450b820a","Type":"ContainerStarted","Data":"88ca662f63a235a6f55dcd3b3d8dfd5b3de712fee6e60ccbee3a45ed68aecd59"}
Oct 31 13:03:11.368742 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.368711    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9" event={"ID":"4504b20c-a7d8-49e9-a572-9004fce4af1d","Type":"ContainerStarted","Data":"d911c55ba548c6c1bd2b4c510c219287e2c4c1402d7a679dc648bfe5e6ffd553"}
Oct 31 13:03:11.370251 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.370201    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w7wlc" event={"ID":"a096aa5d-63cc-41a6-ad78-bc64d2d33e94","Type":"ContainerStarted","Data":"9c9410fc6253e95927cb81986debfac06c5abf46cc78eacce0938c3c9c695500"}
Oct 31 13:03:11.394582 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.393572    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-sqxqp" podStartSLOduration=354.393555256 podStartE2EDuration="5m54.393555256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:11.375450067 +0000 UTC m=+33.848448284" watchObservedRunningTime="2026-10-31 13:03:11.393555256 +0000 UTC m=+33.866553490"
Oct 31 13:03:11.413207 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.413156    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wvdhx" podStartSLOduration=354.413139961 podStartE2EDuration="5m54.413139961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:11.394450957 +0000 UTC m=+33.867449175" watchObservedRunningTime="2026-10-31 13:03:11.413139961 +0000 UTC m=+33.886138180"
Oct 31 13:03:11.724547 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.723082    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:11.724547 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.723169    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:11.724547 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.723202    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:11.724547 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.723241    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:11.724547 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.723272    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:11.724547 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.723345    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:11.724547 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.723414    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:11.724547 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.723550    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 13:03:11.724547 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.723610    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs podName:cf350772-aff8-4051-820a-dd114e01e441 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:43.723592856 +0000 UTC m=+66.196591065 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs") pod "network-metrics-daemon-pphvg" (UID: "cf350772-aff8-4051-820a-dd114e01e441") : secret "metrics-daemon-secret" not found
Oct 31 13:03:11.724547 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.723971    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:11.724547 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.723988    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76fcd8cbb5-tmzzx: secret "image-registry-tls" not found
Oct 31 13:03:11.724547 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.724035    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls podName:7620273c-a4f1-4e1f-ba26-453913b2e171 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:13.724020272 +0000 UTC m=+36.197018482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls") pod "image-registry-76fcd8cbb5-tmzzx" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171") : secret "image-registry-tls" not found
Oct 31 13:03:11.724547 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.724102    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 13:03:11.724547 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.724134    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:13.724123092 +0000 UTC m=+36.197121302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : secret "router-metrics-certs-default" not found
Oct 31 13:03:11.724547 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.724192    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 13:03:11.724547 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.724220    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls podName:7810b5de-3907-4a6e-b11e-0c631c148438 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:13.72421039 +0000 UTC m=+36.197208599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7sfns" (UID: "7810b5de-3907-4a6e-b11e-0c631c148438") : secret "samples-operator-tls" not found
Oct 31 13:03:11.725652 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.724270    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 13:03:11.725652 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.724314    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert podName:73811006-7011-4339-800d-5f1b8a9bbf88 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:13.724288094 +0000 UTC m=+36.197286302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-swrc7" (UID: "73811006-7011-4339-800d-5f1b8a9bbf88") : secret "networking-console-plugin-cert" not found
Oct 31 13:03:11.725652 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.724415    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:13.724405066 +0000 UTC m=+36.197403266 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : configmap references non-existent config key: service-ca.crt
Oct 31 13:03:11.725652 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.724474    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:11.725652 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.724506    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls podName:bf8d1062-0bb5-46f2-98bf-e4d52689294c nodeName:}" failed. No retries permitted until 2026-10-31 13:03:13.724494316 +0000 UTC m=+36.197492524 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-44698" (UID: "bf8d1062-0bb5-46f2-98bf-e4d52689294c") : secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:11.824278 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.824248    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:11.833624 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.833560    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:11.925452 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.925413    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:11.925617 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.925506    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7sk\" (UniqueName: \"kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk\") pod \"network-check-target-tkc46\" (UID: \"209e1f84-8138-42c3-bd7a-5d1bba5457d0\") " pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:03:11.925617 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.925541    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:11.925800 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.925784    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 13:03:11.925863 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.925844    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls podName:bdce0f75-852b-49b6-9da3-f662bb82bf94 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:13.925827319 +0000 UTC m=+36.398825520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls") pod "dns-default-pztwr" (UID: "bdce0f75-852b-49b6-9da3-f662bb82bf94") : secret "dns-default-metrics-tls" not found
Oct 31 13:03:11.926255 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.926237    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:11.926353 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.926257    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694db45b8d-szb8q: secret "image-registry-tls" not found
Oct 31 13:03:11.926353 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:11.926330    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls podName:298ad120-4782-4b13-985f-5af7fb24d97d nodeName:}" failed. No retries permitted until 2026-10-31 13:03:13.926313848 +0000 UTC m=+36.399312047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls") pod "image-registry-694db45b8d-szb8q" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d") : secret "image-registry-tls" not found
Oct 31 13:03:11.937332 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.935814    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv7sk\" (UniqueName: \"kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk\") pod \"network-check-target-tkc46\" (UID: \"209e1f84-8138-42c3-bd7a-5d1bba5457d0\") " pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:03:11.986989 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:11.986925    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:03:12.005975 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:12.005918    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:12.307577 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:12.306876    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-dswrj" podStartSLOduration=355.306860626 podStartE2EDuration="5m55.306860626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:11.414264401 +0000 UTC m=+33.887262640" watchObservedRunningTime="2026-10-31 13:03:12.306860626 +0000 UTC m=+34.779858842"
Oct 31 13:03:12.315318 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:12.308794    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrj9c"]
Oct 31 13:03:12.320018 ip-10-0-137-77 kubenswrapper[2571]: W1031 13:03:12.319991    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91204e03_5b1b_4084_a699_ac7b59db4be5.slice/crio-d39cc227edf77ab01436fdd077d8415a9a47ee6a97b912072628557ae181a5d3 WatchSource:0}: Error finding container d39cc227edf77ab01436fdd077d8415a9a47ee6a97b912072628557ae181a5d3: Status 404 returned error can't find the container with id d39cc227edf77ab01436fdd077d8415a9a47ee6a97b912072628557ae181a5d3
Oct 31 13:03:12.329068 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:12.326659    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tkc46"]
Oct 31 13:03:12.379195 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:12.379073    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tkc46" event={"ID":"209e1f84-8138-42c3-bd7a-5d1bba5457d0","Type":"ContainerStarted","Data":"38fcb4fc076605b04085b272d1f7f926f064ed61bb8405313fbbd8fd9a304031"}
Oct 31 13:03:12.383236 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:12.382283    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrj9c" event={"ID":"91204e03-5b1b-4084-a699-ac7b59db4be5","Type":"ContainerStarted","Data":"d39cc227edf77ab01436fdd077d8415a9a47ee6a97b912072628557ae181a5d3"}
Oct 31 13:03:13.388492 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:13.387661    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w7wlc" event={"ID":"a096aa5d-63cc-41a6-ad78-bc64d2d33e94","Type":"ContainerStarted","Data":"cda4fd2a9a2535275a1c1d6869c2b81bcba2d16a0cd3cfcf603b826c7df466e7"}
Oct 31 13:03:13.747005 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:13.746969    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:13.747185 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:13.747015    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:13.747185 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:13.747040    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:13.747185 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:13.747076    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:13.747185 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:13.747148    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:13.747185 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:13.747171    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:13.747475 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.747280    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 13:03:13.747475 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.747357    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:17.747338555 +0000 UTC m=+40.220336751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : secret "router-metrics-certs-default" not found
Oct 31 13:03:13.748060 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.748039    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 13:03:13.748156 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.748103    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls podName:7810b5de-3907-4a6e-b11e-0c631c148438 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:17.748087497 +0000 UTC m=+40.221085694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7sfns" (UID: "7810b5de-3907-4a6e-b11e-0c631c148438") : secret "samples-operator-tls" not found
Oct 31 13:03:13.748323 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.748162    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 13:03:13.748323 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.748202    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert podName:73811006-7011-4339-800d-5f1b8a9bbf88 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:17.74818911 +0000 UTC m=+40.221187322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-swrc7" (UID: "73811006-7011-4339-800d-5f1b8a9bbf88") : secret "networking-console-plugin-cert" not found
Oct 31 13:03:13.748323 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.748266    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:17.748256325 +0000 UTC m=+40.221254524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : configmap references non-existent config key: service-ca.crt
Oct 31 13:03:13.748559 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.748332    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:13.748559 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.748366    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls podName:bf8d1062-0bb5-46f2-98bf-e4d52689294c nodeName:}" failed. No retries permitted until 2026-10-31 13:03:17.748354099 +0000 UTC m=+40.221352296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-44698" (UID: "bf8d1062-0bb5-46f2-98bf-e4d52689294c") : secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:13.748559 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.748432    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:13.748559 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.748444    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76fcd8cbb5-tmzzx: secret "image-registry-tls" not found
Oct 31 13:03:13.748559 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.748475    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls podName:7620273c-a4f1-4e1f-ba26-453913b2e171 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:17.748465404 +0000 UTC m=+40.221463601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls") pod "image-registry-76fcd8cbb5-tmzzx" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171") : secret "image-registry-tls" not found
Oct 31 13:03:13.949748 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:13.949657    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:13.949916 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:13.949758    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:13.950014 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.949996    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 13:03:13.950088 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.950076    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls podName:bdce0f75-852b-49b6-9da3-f662bb82bf94 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:17.950057947 +0000 UTC m=+40.423056158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls") pod "dns-default-pztwr" (UID: "bdce0f75-852b-49b6-9da3-f662bb82bf94") : secret "dns-default-metrics-tls" not found
Oct 31 13:03:13.950549 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.950527    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:13.950635 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.950552    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694db45b8d-szb8q: secret "image-registry-tls" not found
Oct 31 13:03:13.950635 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:13.950600    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls podName:298ad120-4782-4b13-985f-5af7fb24d97d nodeName:}" failed. No retries permitted until 2026-10-31 13:03:17.950582246 +0000 UTC m=+40.423580441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls") pod "image-registry-694db45b8d-szb8q" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d") : secret "image-registry-tls" not found
Oct 31 13:03:16.421325 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:16.420052    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-cpk9z" event={"ID":"2a1a0fc2-36f2-4635-97b6-97faed580133","Type":"ContainerStarted","Data":"e29969cb3c7add6009b36661474b30c30d52f11df4289aea516bfb37a2d4efc6"}
Oct 31 13:03:16.431629 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:16.431323    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s" event={"ID":"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a","Type":"ContainerStarted","Data":"14d3d53a0676ca2730194ce8c489da3ac30b0f07c888517ec36651cbd182493d"}
Oct 31 13:03:16.435828 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:16.435364    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft" event={"ID":"b3bc5c9b-54fb-477f-8c98-9884b1b2e159","Type":"ContainerStarted","Data":"42301032689b3a980e66fd0ef98156a04942ebeaa9d862a7ef596d1ae4a03fa8"}
Oct 31 13:03:16.439189 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:16.438581    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/0.log"
Oct 31 13:03:16.439189 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:16.438629    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9" event={"ID":"4504b20c-a7d8-49e9-a572-9004fce4af1d","Type":"ContainerStarted","Data":"d31b551e5cdf30ff8d46f075165d5fa1027bedaeab3f9f46cbd707c9ba2ff28e"}
Oct 31 13:03:16.439189 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:16.438890    2571 scope.go:122] "RemoveContainer" containerID="d31b551e5cdf30ff8d46f075165d5fa1027bedaeab3f9f46cbd707c9ba2ff28e"
Oct 31 13:03:16.443458 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:16.443393    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w7wlc" podStartSLOduration=327.993781797 podStartE2EDuration="5m30.443377496s" totalImagesPullingTime="2.449595699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:13.407950655 +0000 UTC m=+35.880948896" watchObservedRunningTime="2026-10-31 13:03:16.443377496 +0000 UTC m=+38.916375717"
Oct 31 13:03:16.444258 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:16.444181    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-cpk9z" podStartSLOduration=324.87953915 podStartE2EDuration="5m30.44417167s" totalImagesPullingTime="5.56463252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:16.440705722 +0000 UTC m=+38.913704150" watchObservedRunningTime="2026-10-31 13:03:16.44417167 +0000 UTC m=+38.917169891"
Oct 31 13:03:16.485602 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:16.485554    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft" podStartSLOduration=323.987916748 podStartE2EDuration="5m29.485536458s" totalImagesPullingTime="5.49761971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:16.483158672 +0000 UTC m=+38.956156891" watchObservedRunningTime="2026-10-31 13:03:16.485536458 +0000 UTC m=+38.958534684"
Oct 31 13:03:16.506228 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:16.504856    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s" podStartSLOduration=324.068108721 podStartE2EDuration="5m29.504840722s" totalImagesPullingTime="5.436732001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:16.504382375 +0000 UTC m=+38.977380610" watchObservedRunningTime="2026-10-31 13:03:16.504840722 +0000 UTC m=+38.977838942"
Oct 31 13:03:17.274619 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.274548    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9hjjp"]
Oct 31 13:03:17.296426 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.296389    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:17.302644 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.302618    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9hjjp"]
Oct 31 13:03:17.386919 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.386888    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aab0721f-f591-4a46-b56e-27848f31642f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:17.387046 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.387008    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aab0721f-f591-4a46-b56e-27848f31642f-data-volume\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:17.387119 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.387051    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpzp2\" (UniqueName: \"kubernetes.io/projected/aab0721f-f591-4a46-b56e-27848f31642f-kube-api-access-jpzp2\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:17.387181 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.387140    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aab0721f-f591-4a46-b56e-27848f31642f-crio-socket\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:17.387253 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.387234    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:17.391991 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.391952    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 13:03:17.392199 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.392174    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tmfz7\""
Oct 31 13:03:17.398076 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.398042    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 13:03:17.445097 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.445012    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/1.log"
Oct 31 13:03:17.445593 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.445541    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/0.log"
Oct 31 13:03:17.445593 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.445572    2571 generic.go:356] "Generic (PLEG): container finished" podID="4504b20c-a7d8-49e9-a572-9004fce4af1d" containerID="d31b551e5cdf30ff8d46f075165d5fa1027bedaeab3f9f46cbd707c9ba2ff28e" exitCode=255
Oct 31 13:03:17.445593 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.445585    2571 generic.go:356] "Generic (PLEG): container finished" podID="4504b20c-a7d8-49e9-a572-9004fce4af1d" containerID="3d8b55e30b935d5d939f5ab6056e8c94bd638608bc9e562793f30465208bd095" exitCode=255
Oct 31 13:03:17.445742 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.445600    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9" event={"ID":"4504b20c-a7d8-49e9-a572-9004fce4af1d","Type":"ContainerDied","Data":"d31b551e5cdf30ff8d46f075165d5fa1027bedaeab3f9f46cbd707c9ba2ff28e"}
Oct 31 13:03:17.445742 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.445647    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9" event={"ID":"4504b20c-a7d8-49e9-a572-9004fce4af1d","Type":"ContainerDied","Data":"3d8b55e30b935d5d939f5ab6056e8c94bd638608bc9e562793f30465208bd095"}
Oct 31 13:03:17.445742 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.445668    2571 scope.go:122] "RemoveContainer" containerID="d31b551e5cdf30ff8d46f075165d5fa1027bedaeab3f9f46cbd707c9ba2ff28e"
Oct 31 13:03:17.445956 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.445938    2571 scope.go:122] "RemoveContainer" containerID="3d8b55e30b935d5d939f5ab6056e8c94bd638608bc9e562793f30465208bd095"
Oct 31 13:03:17.446208 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.446180    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7v9w9_openshift-console-operator(4504b20c-a7d8-49e9-a572-9004fce4af1d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9" podUID="4504b20c-a7d8-49e9-a572-9004fce4af1d"
Oct 31 13:03:17.448219 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.448163    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l42ds" event={"ID":"bd718975-d0cb-4a21-8383-c29164ae4f31","Type":"ContainerStarted","Data":"4c7a8ae83e0984f0c93b36c93d43aa627632d6c37984ae280c69fd58504e3490"}
Oct 31 13:03:17.475392 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.475371    2571 scope.go:122] "RemoveContainer" containerID="d31b551e5cdf30ff8d46f075165d5fa1027bedaeab3f9f46cbd707c9ba2ff28e"
Oct 31 13:03:17.475741 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.475676    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d31b551e5cdf30ff8d46f075165d5fa1027bedaeab3f9f46cbd707c9ba2ff28e\": container with ID starting with d31b551e5cdf30ff8d46f075165d5fa1027bedaeab3f9f46cbd707c9ba2ff28e not found: ID does not exist" containerID="d31b551e5cdf30ff8d46f075165d5fa1027bedaeab3f9f46cbd707c9ba2ff28e"
Oct 31 13:03:17.475741 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.475707    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d31b551e5cdf30ff8d46f075165d5fa1027bedaeab3f9f46cbd707c9ba2ff28e"} err="failed to get container status \"d31b551e5cdf30ff8d46f075165d5fa1027bedaeab3f9f46cbd707c9ba2ff28e\": rpc error: code = NotFound desc = could not find container \"d31b551e5cdf30ff8d46f075165d5fa1027bedaeab3f9f46cbd707c9ba2ff28e\": container with ID starting with d31b551e5cdf30ff8d46f075165d5fa1027bedaeab3f9f46cbd707c9ba2ff28e not found: ID does not exist"
Oct 31 13:03:17.488480 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.488453    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aab0721f-f591-4a46-b56e-27848f31642f-data-volume\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:17.488597 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.488491    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpzp2\" (UniqueName: \"kubernetes.io/projected/aab0721f-f591-4a46-b56e-27848f31642f-kube-api-access-jpzp2\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:17.488669 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.488607    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aab0721f-f591-4a46-b56e-27848f31642f-crio-socket\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:17.488727 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.488701    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:17.488779 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.488765    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aab0721f-f591-4a46-b56e-27848f31642f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:17.488849 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.488837    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 13:03:17.488897 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.488881    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aab0721f-f591-4a46-b56e-27848f31642f-kube-rbac-proxy-cm podName:aab0721f-f591-4a46-b56e-27848f31642f nodeName:}" failed. No retries permitted until 2026-10-31 13:03:17.988869939 +0000 UTC m=+40.461868134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aab0721f-f591-4a46-b56e-27848f31642f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9hjjp" (UID: "aab0721f-f591-4a46-b56e-27848f31642f") : configmap "kube-rbac-proxy" not found
Oct 31 13:03:17.489866 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.489844    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aab0721f-f591-4a46-b56e-27848f31642f-crio-socket\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:17.490316 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.490143    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aab0721f-f591-4a46-b56e-27848f31642f-data-volume\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:17.491021 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.491001    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:17.491109 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.491060    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls podName:aab0721f-f591-4a46-b56e-27848f31642f nodeName:}" failed. No retries permitted until 2026-10-31 13:03:17.991043633 +0000 UTC m=+40.464041835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hjjp" (UID: "aab0721f-f591-4a46-b56e-27848f31642f") : secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:17.513597 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.513574    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpzp2\" (UniqueName: \"kubernetes.io/projected/aab0721f-f591-4a46-b56e-27848f31642f-kube-api-access-jpzp2\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:17.791847 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.791808    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:17.791999 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.791871    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:17.791999 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.791913    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:17.791999 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.791934    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:17.791999 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.791973    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 13:03:17.792197 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.792005    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 13:03:17.792197 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.792037    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:17.792197 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.791980    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:17.792197 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.791977    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 13:03:17.792197 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.792052    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.792029472 +0000 UTC m=+48.265027693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : secret "router-metrics-certs-default" not found
Oct 31 13:03:17.792197 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.792106    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert podName:73811006-7011-4339-800d-5f1b8a9bbf88 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.792092887 +0000 UTC m=+48.265091082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-swrc7" (UID: "73811006-7011-4339-800d-5f1b8a9bbf88") : secret "networking-console-plugin-cert" not found
Oct 31 13:03:17.792197 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.792123    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.792116812 +0000 UTC m=+48.265115006 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : configmap references non-existent config key: service-ca.crt
Oct 31 13:03:17.792197 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.792141    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls podName:bf8d1062-0bb5-46f2-98bf-e4d52689294c nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.792132908 +0000 UTC m=+48.265131103 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-44698" (UID: "bf8d1062-0bb5-46f2-98bf-e4d52689294c") : secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:17.792197 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.792152    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls podName:7810b5de-3907-4a6e-b11e-0c631c148438 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.792147316 +0000 UTC m=+48.265145510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7sfns" (UID: "7810b5de-3907-4a6e-b11e-0c631c148438") : secret "samples-operator-tls" not found
Oct 31 13:03:17.792587 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.792254    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:17.792587 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.792364    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:17.792587 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.792379    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76fcd8cbb5-tmzzx: secret "image-registry-tls" not found
Oct 31 13:03:17.792587 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.792408    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls podName:7620273c-a4f1-4e1f-ba26-453913b2e171 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.792398432 +0000 UTC m=+48.265396655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls") pod "image-registry-76fcd8cbb5-tmzzx" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171") : secret "image-registry-tls" not found
Oct 31 13:03:17.995154 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.993988    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:17.995154 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.994130    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:17.995154 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.994204    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aab0721f-f591-4a46-b56e-27848f31642f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:17.995154 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:17.994242    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:17.995154 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.994429    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:17.995154 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.994444    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694db45b8d-szb8q: secret "image-registry-tls" not found
Oct 31 13:03:17.995154 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.994503    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls podName:298ad120-4782-4b13-985f-5af7fb24d97d nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.994485718 +0000 UTC m=+48.467483918 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls") pod "image-registry-694db45b8d-szb8q" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d") : secret "image-registry-tls" not found
Oct 31 13:03:17.995154 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.994891    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 13:03:17.995154 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.994934    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls podName:bdce0f75-852b-49b6-9da3-f662bb82bf94 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.994920843 +0000 UTC m=+48.467919046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls") pod "dns-default-pztwr" (UID: "bdce0f75-852b-49b6-9da3-f662bb82bf94") : secret "dns-default-metrics-tls" not found
Oct 31 13:03:17.995154 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.994989    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:17.995154 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.995022    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls podName:aab0721f-f591-4a46-b56e-27848f31642f nodeName:}" failed. No retries permitted until 2026-10-31 13:03:18.995011224 +0000 UTC m=+41.468009421 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hjjp" (UID: "aab0721f-f591-4a46-b56e-27848f31642f") : secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:17.995154 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.995053    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 13:03:17.995154 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:17.995083    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aab0721f-f591-4a46-b56e-27848f31642f-kube-rbac-proxy-cm podName:aab0721f-f591-4a46-b56e-27848f31642f nodeName:}" failed. No retries permitted until 2026-10-31 13:03:18.995074548 +0000 UTC m=+41.468072758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aab0721f-f591-4a46-b56e-27848f31642f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9hjjp" (UID: "aab0721f-f591-4a46-b56e-27848f31642f") : configmap "kube-rbac-proxy" not found
Oct 31 13:03:18.082017 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:18.081983    2571 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/systemd-update-utmp-runlevel.service\": RecentStats: unable to find data in memory cache]"
Oct 31 13:03:18.453614 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:18.453393    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lcmgp" event={"ID":"49b449ac-b419-4bd1-9896-0941450b820a","Type":"ContainerStarted","Data":"0aad34dc013c2cad06f197d3966d21d4f74a5ca2f05ed2bf30d7b1a35b3e9849"}
Oct 31 13:03:18.455049 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:18.455027    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/1.log"
Oct 31 13:03:18.455396 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:18.455373    2571 scope.go:122] "RemoveContainer" containerID="3d8b55e30b935d5d939f5ab6056e8c94bd638608bc9e562793f30465208bd095"
Oct 31 13:03:18.455571 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:18.455550    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7v9w9_openshift-console-operator(4504b20c-a7d8-49e9-a572-9004fce4af1d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9" podUID="4504b20c-a7d8-49e9-a572-9004fce4af1d"
Oct 31 13:03:18.500336 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:18.499245    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lcmgp" podStartSLOduration=323.529129674 podStartE2EDuration="5m30.499227335s" totalImagesPullingTime="6.970097661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:18.476497485 +0000 UTC m=+40.949495734" watchObservedRunningTime="2026-10-31 13:03:18.499227335 +0000 UTC m=+40.972225553"
Oct 31 13:03:18.525229 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:18.525177    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l42ds" podStartSLOduration=3.0225704 podStartE2EDuration="9.525160769s" totalImagesPullingTime="6.502590369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:18.500454695 +0000 UTC m=+40.973452938" watchObservedRunningTime="2026-10-31 13:03:18.525160769 +0000 UTC m=+40.998158988"
Oct 31 13:03:19.004394 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:19.004352    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:19.004592 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:19.004451    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aab0721f-f591-4a46-b56e-27848f31642f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:19.004592 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:19.004522    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:19.004702 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:19.004595    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls podName:aab0721f-f591-4a46-b56e-27848f31642f nodeName:}" failed. No retries permitted until 2026-10-31 13:03:21.00457475 +0000 UTC m=+43.477572964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hjjp" (UID: "aab0721f-f591-4a46-b56e-27848f31642f") : secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:19.004702 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:19.004636    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 13:03:19.004702 ip-10-0-137-77 kubenswrapper[2571]: E1031 13:03:19.004690    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aab0721f-f591-4a46-b56e-27848f31642f-kube-rbac-proxy-cm podName:aab0721f-f591-4a46-b56e-27848f31642f nodeName:}" failed. No retries permitted until 2026-10-31 13:03:21.004675156 +0000 UTC m=+43.477673351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aab0721f-f591-4a46-b56e-27848f31642f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9hjjp" (UID: "aab0721f-f591-4a46-b56e-27848f31642f") : configmap "kube-rbac-proxy" not found
Oct 31 13:03:19.462201 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:19.462160    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tkc46" event={"ID":"209e1f84-8138-42c3-bd7a-5d1bba5457d0","Type":"ContainerStarted","Data":"d13519e9b2ded96f5d72d83ffb698a845f01f4ac4dbe11c7ca7034947fec782f"}
Oct 31 13:03:19.482541 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:19.482495    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tkc46" podStartSLOduration=35.373075045 podStartE2EDuration="41.482482641s" totalImagesPullingTime="6.109407596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:19.481361922 +0000 UTC m=+41.954360140" watchObservedRunningTime="2026-10-31 13:03:19.482482641 +0000 UTC m=+41.955480857"
Oct 31 13:03:20.297390 ip-10-0-137-77 kubenswrapper[2571]: I1031 13:03:20.297271    2571 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 13:03:20.297531 ip-10-0-137-77 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 13:03:20.325918 ip-10-0-137-77 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 13:03:20.326083 ip-10-0-137-77 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 13:03:20.326243 ip-10-0-137-77 systemd[1]: kubelet.service: Consumed 2.971s CPU time.
Oct 31 13:03:20.339792 ip-10-0-137-77 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 13:03:20.434953 ip-10-0-137-77 kubenswrapper[4660]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 13:03:20.434953 ip-10-0-137-77 kubenswrapper[4660]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 13:03:20.434953 ip-10-0-137-77 kubenswrapper[4660]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 13:03:20.434953 ip-10-0-137-77 kubenswrapper[4660]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 13:03:20.437221 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437140    4660 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:03:20.437221 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437217    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:03:20.437221 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437223    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:03:20.437221 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437227    4660 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437231    4660 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437235    4660 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437239    4660 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437243    4660 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437247    4660 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437251    4660 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437254    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437258    4660 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437263    4660 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437267    4660 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437271    4660 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437275    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437279    4660 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437283    4660 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437287    4660 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437308    4660 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437312    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437316    4660 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437330    4660 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:03:20.437501 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437334    4660 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437338    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437342    4660 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437345    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437349    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437354    4660 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437358    4660 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437361    4660 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437365    4660 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437370    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437377    4660 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437383    4660 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437387    4660 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437392    4660 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437397    4660 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437402    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437406    4660 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437410    4660 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437415    4660 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:03:20.438395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437420    4660 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437425    4660 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437429    4660 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437434    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437438    4660 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437442    4660 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437447    4660 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437451    4660 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437455    4660 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437459    4660 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437464    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437471    4660 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437478    4660 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437483    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437488    4660 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437493    4660 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437497    4660 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437501    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437505    4660 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437510    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:03:20.439207 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437514    4660 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437518    4660 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437522    4660 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437527    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437531    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437536    4660 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437540    4660 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437544    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437548    4660 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437553    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437558    4660 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437563    4660 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437567    4660 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437571    4660 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437575    4660 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437580    4660 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437584    4660 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437588    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437593    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:03:20.440092 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437597    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437601    4660 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437605    4660 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437610    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437614    4660 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437618    4660 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437622    4660 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437629    4660 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437634    4660 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437638    4660 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437643    4660 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437647    4660 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437651    4660 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437655    4660 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437659    4660 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437663    4660 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437668    4660 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437672    4660 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437677    4660 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437681    4660 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:03:20.440929 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437685    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437689    4660 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437693    4660 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437697    4660 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.437702    4660 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.438956    4660 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.438966    4660 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.438970    4660 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.438975    4660 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.438979    4660 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.438984    4660 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.438989    4660 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.438993    4660 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.438997    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439002    4660 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439006    4660 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439010    4660 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439015    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439019    4660 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439024    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:03:20.441835 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439029    4660 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439033    4660 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439037    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439042    4660 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439046    4660 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439050    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439054    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439058    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439063    4660 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439067    4660 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439071    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439075    4660 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439079    4660 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439083    4660 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439088    4660 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439092    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439096    4660 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439101    4660 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439105    4660 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439110    4660 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:03:20.442749 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439114    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439118    4660 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439123    4660 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439127    4660 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439132    4660 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439136    4660 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439140    4660 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439144    4660 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439148    4660 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439152    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439157    4660 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439163    4660 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439169    4660 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439176    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439181    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439186    4660 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439190    4660 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439194    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439198    4660 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:03:20.443582 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439202    4660 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439206    4660 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439210    4660 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439214    4660 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439217    4660 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439221    4660 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439225    4660 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439229    4660 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439233    4660 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439238    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439242    4660 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439248    4660 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439252    4660 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439256    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439260    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439265    4660 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439269    4660 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439276    4660 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439282    4660 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:03:20.444182 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439287    4660 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439309    4660 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439314    4660 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439318    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439323    4660 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439327    4660 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439331    4660 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439334    4660 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439339    4660 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439343    4660 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439347    4660 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439351    4660 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439355    4660 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439359    4660 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439364    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439368    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439372    4660 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439376    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439380    4660 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439385    4660 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:03:20.444715 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439389    4660 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439393    4660 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439398    4660 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439404    4660 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439409    4660 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439413    4660 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439417    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439422    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439426    4660 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439430    4660 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439434    4660 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439438    4660 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.439442    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440211    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440220    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440252    4660 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440257    4660 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440261    4660 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440266    4660 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440270    4660 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:03:20.445202 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440274    4660 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440279    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440283    4660 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440287    4660 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440307    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440313    4660 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440318    4660 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440322    4660 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440326    4660 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440331    4660 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440335    4660 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440339    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440344    4660 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440348    4660 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440353    4660 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440357    4660 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440361    4660 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440367    4660 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440371    4660 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:03:20.445694 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440376    4660 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440380    4660 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440384    4660 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440388    4660 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440392    4660 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440397    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440401    4660 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440406    4660 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440410    4660 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440415    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440419    4660 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440423    4660 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440427    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440431    4660 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440436    4660 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440440    4660 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440444    4660 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440449    4660 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440453    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:03:20.446199 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440457    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440461    4660 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440465    4660 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440470    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440474    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440482    4660 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440486    4660 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440491    4660 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440495    4660 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440499    4660 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440503    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440508    4660 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440513    4660 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440517    4660 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440521    4660 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440525    4660 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440530    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440534    4660 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440538    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:03:20.446732 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440543    4660 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440547    4660 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440554    4660 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440560    4660 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440565    4660 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440571    4660 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440576    4660 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440580    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440584    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440588    4660 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440593    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440597    4660 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440601    4660 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440605    4660 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440609    4660 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440614    4660 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440618    4660 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440622    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440628    4660 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:03:20.447206 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440632    4660 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440636    4660 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440641    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440645    4660 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440650    4660 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440655    4660 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440659    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440664    4660 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440668    4660 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440673    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440677    4660 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440681    4660 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440686    4660 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440690    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440694    4660 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440699    4660 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440703    4660 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440707    4660 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440713    4660 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440718    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:03:20.447685 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440722    4660 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440727    4660 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.440734    4660 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440867    4660 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440878    4660 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440887    4660 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440894    4660 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440901    4660 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440907    4660 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440913    4660 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440919    4660 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440926    4660 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440931    4660 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440936    4660 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440941    4660 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440946    4660 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440951    4660 flags.go:64] FLAG: --cgroup-root=""
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440956    4660 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440961    4660 flags.go:64] FLAG: --client-ca-file=""
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440966    4660 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440972    4660 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440980    4660 flags.go:64] FLAG: --cluster-domain=""
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440984    4660 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440989    4660 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 13:03:20.448161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.440994    4660 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441000    4660 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441006    4660 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441011    4660 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441016    4660 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441022    4660 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441026    4660 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441031    4660 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441036    4660 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441041    4660 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441045    4660 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441052    4660 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441057    4660 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441062    4660 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441067    4660 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441072    4660 flags.go:64] FLAG: --enable-server="true"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441076    4660 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441083    4660 flags.go:64] FLAG: --event-burst="100"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441088    4660 flags.go:64] FLAG: --event-qps="50"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441094    4660 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441099    4660 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441104    4660 flags.go:64] FLAG: --eviction-hard=""
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441110    4660 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441116    4660 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441120    4660 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 13:03:20.449247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441125    4660 flags.go:64] FLAG: --eviction-soft=""
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441130    4660 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441135    4660 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441140    4660 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441145    4660 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441150    4660 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441155    4660 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441160    4660 flags.go:64] FLAG: --feature-gates=""
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441170    4660 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441175    4660 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441180    4660 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441185    4660 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441190    4660 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441195    4660 flags.go:64] FLAG: --help="false"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441200    4660 flags.go:64] FLAG: --hostname-override="ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441205    4660 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441210    4660 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441215    4660 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441221    4660 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441226    4660 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441231    4660 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441236    4660 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441241    4660 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 13:03:20.449912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441245    4660 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441250    4660 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441255    4660 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441262    4660 flags.go:64] FLAG: --kube-reserved=""
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441267    4660 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441272    4660 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441277    4660 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441281    4660 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441286    4660 flags.go:64] FLAG: --lock-file=""
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441306    4660 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441311    4660 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441316    4660 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441325    4660 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441329    4660 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441335    4660 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441340    4660 flags.go:64] FLAG: --logging-format="text"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441345    4660 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441350    4660 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441355    4660 flags.go:64] FLAG: --manifest-url=""
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441359    4660 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441366    4660 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441372    4660 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441378    4660 flags.go:64] FLAG: --max-pods="110"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441383    4660 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441388    4660 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 13:03:20.450493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441392    4660 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441397    4660 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441402    4660 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441406    4660 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441411    4660 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441422    4660 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441427    4660 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441432    4660 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441437    4660 flags.go:64] FLAG: --pod-cidr=""
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441441    4660 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441448    4660 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441453    4660 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441457    4660 flags.go:64] FLAG: --port="10250"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441462    4660 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441467    4660 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0d4d27fe16577aee8"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441472    4660 flags.go:64] FLAG: --qos-reserved=""
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441477    4660 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441482    4660 flags.go:64] FLAG: --register-node="true"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441487    4660 flags.go:64] FLAG: --register-with-taints=""
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441492    4660 flags.go:64] FLAG: --registry-burst="10"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441497    4660 flags.go:64] FLAG: --registry-qps="5"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441502    4660 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441509    4660 flags.go:64] FLAG: --reserved-memory=""
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441515    4660 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441520    4660 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441525    4660 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 13:03:20.451133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441530    4660 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441536    4660 flags.go:64] FLAG: --runonce="false"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441541    4660 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441547    4660 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441551    4660 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441556    4660 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441561    4660 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441566    4660 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441570    4660 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441575    4660 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441580    4660 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441584    4660 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441589    4660 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441594    4660 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441598    4660 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441603    4660 flags.go:64] FLAG: --system-cgroups=""
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441610    4660 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441618    4660 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441622    4660 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441627    4660 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441634    4660 flags.go:64] FLAG: --tls-min-version=""
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441639    4660 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441644    4660 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441648    4660 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441653    4660 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 13:03:20.451864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441658    4660 flags.go:64] FLAG: --v="2"
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441665    4660 flags.go:64] FLAG: --version="false"
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441671    4660 flags.go:64] FLAG: --vmodule=""
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441677    4660 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.441683    4660 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441898    4660 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441907    4660 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441912    4660 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441921    4660 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441926    4660 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441932    4660 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441936    4660 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441941    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441945    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441950    4660 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441954    4660 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441959    4660 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441963    4660 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441967    4660 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441972    4660 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:03:20.452937 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441976    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441980    4660 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441984    4660 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441990    4660 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441995    4660 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.441999    4660 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442005    4660 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442012    4660 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442018    4660 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442022    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442026    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442030    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442037    4660 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442067    4660 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442072    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442077    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442081    4660 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442087    4660 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442092    4660 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:03:20.453514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442096    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442102    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442107    4660 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442111    4660 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442115    4660 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442119    4660 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442123    4660 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442128    4660 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442132    4660 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442136    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442140    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442145    4660 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442149    4660 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442153    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442157    4660 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442161    4660 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442168    4660 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442172    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442177    4660 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442181    4660 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:03:20.453988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442186    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442190    4660 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442194    4660 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442198    4660 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442202    4660 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442207    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442211    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442215    4660 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442219    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442224    4660 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442228    4660 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442233    4660 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442237    4660 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442243    4660 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442247    4660 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442252    4660 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442256    4660 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442261    4660 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442265    4660 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442269    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:03:20.454498 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442273    4660 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442277    4660 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442281    4660 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442286    4660 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442305    4660 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442309    4660 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442314    4660 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442318    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442324    4660 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442328    4660 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442333    4660 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442337    4660 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442341    4660 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442346    4660 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442350    4660 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442354    4660 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442358    4660 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442362    4660 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442366    4660 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442371    4660 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:03:20.455046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442376    4660 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:03:20.455670 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442380    4660 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:03:20.455670 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442384    4660 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:03:20.455670 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442388    4660 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:03:20.455670 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442393    4660 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:03:20.455670 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442399    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:03:20.455670 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442404    4660 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:03:20.455670 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442408    4660 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:03:20.455670 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442412    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:03:20.455670 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442416    4660 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:03:20.455670 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442421    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:03:20.455670 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442425    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:03:20.455670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.442468    4660 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 13:03:20.455670 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442523    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:03:20.455670 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442530    4660 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:03:20.455670 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442535    4660 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:03:20.455670 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442541    4660 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442546    4660 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442550    4660 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442554    4660 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442558    4660 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442564    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442569    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442573    4660 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442577    4660 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442581    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442586    4660 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442590    4660 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442594    4660 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442598    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442602    4660 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442607    4660 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442611    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442616    4660 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442620    4660 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442624    4660 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:03:20.456179 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442629    4660 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442636    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442640    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442644    4660 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442648    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442653    4660 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442657    4660 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442661    4660 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442665    4660 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442669    4660 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442673    4660 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442678    4660 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442682    4660 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442686    4660 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442691    4660 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442696    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442700    4660 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442706    4660 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442710    4660 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442715    4660 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:03:20.456767 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442719    4660 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442723    4660 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442728    4660 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442732    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442736    4660 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442740    4660 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442745    4660 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442749    4660 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442752    4660 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442756    4660 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442760    4660 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442764    4660 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442768    4660 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442773    4660 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442778    4660 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442782    4660 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442787    4660 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442791    4660 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442795    4660 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442800    4660 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:03:20.457313 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442804    4660 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442808    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442813    4660 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442817    4660 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442821    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442825    4660 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442830    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442834    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442838    4660 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442847    4660 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442851    4660 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442856    4660 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442860    4660 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442864    4660 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442868    4660 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442872    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442877    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442881    4660 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442885    4660 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:03:20.457866 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442889    4660 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442893    4660 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442898    4660 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442902    4660 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442906    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442911    4660 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442915    4660 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442920    4660 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442925    4660 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442929    4660 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442935    4660 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442941    4660 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442946    4660 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442950    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442954    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442959    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442965    4660 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442971    4660 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442976    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:03:20.458384 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442980    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442985    4660 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442991    4660 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.442998    4660 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.443002    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.448305    4660 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.448320    4660 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.448337    4660 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.448341    4660 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448398    4660 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448403    4660 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448407    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448410    4660 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448413    4660 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448416    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448419    4660 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448422    4660 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448424    4660 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448427    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:03:20.458890 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448430    4660 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448432    4660 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448435    4660 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448438    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448441    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448444    4660 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448446    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448449    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448452    4660 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448454    4660 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448457    4660 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448459    4660 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448462    4660 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448465    4660 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448468    4660 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448470    4660 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448473    4660 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448475    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448478    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:03:20.459365 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448487    4660 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448490    4660 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448493    4660 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448495    4660 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448498    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448500    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448503    4660 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448505    4660 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448508    4660 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448511    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448513    4660 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448516    4660 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448518    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448521    4660 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448523    4660 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448526    4660 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448528    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448531    4660 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448533    4660 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448536    4660 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:03:20.459824 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448538    4660 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448541    4660 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448544    4660 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448547    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448549    4660 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448552    4660 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448554    4660 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448557    4660 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448559    4660 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448562    4660 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448564    4660 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448567    4660 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448570    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448583    4660 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448586    4660 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448588    4660 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448591    4660 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448594    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448598    4660 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:03:20.460342 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448602    4660 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448605    4660 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448608    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448610    4660 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448613    4660 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448616    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448618    4660 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448621    4660 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448624    4660 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448626    4660 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448629    4660 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448632    4660 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448634    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448637    4660 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448639    4660 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448642    4660 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448644    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448647    4660 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448650    4660 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448652    4660 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:03:20.460916 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448656    4660 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448659    4660 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448663    4660 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448666    4660 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448669    4660 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448671    4660 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448674    4660 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448682    4660 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448685    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448688    4660 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448691    4660 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448693    4660 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448696    4660 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448698    4660 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448701    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448703    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448706    4660 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:03:20.461691 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448714    4660 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:03:20.462244 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.448741    4660 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 13:03:20.462244 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448921    4660 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:03:20.462244 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448925    4660 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:03:20.462244 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448928    4660 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:03:20.462244 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448931    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:03:20.462244 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448933    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:03:20.462244 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448936    4660 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:03:20.462244 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448938    4660 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:03:20.462244 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448941    4660 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:03:20.462244 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448945    4660 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:03:20.462244 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448949    4660 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:03:20.462244 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448951    4660 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:03:20.462244 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448954    4660 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:03:20.462244 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448956    4660 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:03:20.462244 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448959    4660 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448961    4660 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448964    4660 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448966    4660 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448969    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448971    4660 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448974    4660 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448976    4660 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448984    4660 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448987    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448990    4660 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448992    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448995    4660 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.448997    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449000    4660 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449002    4660 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449005    4660 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449007    4660 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449010    4660 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449012    4660 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:03:20.462753 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449015    4660 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449017    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449020    4660 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449022    4660 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449024    4660 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449027    4660 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449029    4660 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449032    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449034    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449037    4660 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449039    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449042    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449044    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449050    4660 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449052    4660 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449055    4660 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449057    4660 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449060    4660 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449063    4660 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449065    4660 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:03:20.463239 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449068    4660 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449093    4660 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449098    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449101    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449104    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449107    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449109    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449112    4660 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449115    4660 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449117    4660 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449120    4660 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449122    4660 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449125    4660 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449128    4660 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449130    4660 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449133    4660 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449135    4660 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449138    4660 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449140    4660 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:03:20.463730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449143    4660 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449146    4660 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449148    4660 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449151    4660 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449153    4660 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449156    4660 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449160    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449163    4660 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449165    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449168    4660 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449171    4660 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449173    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449176    4660 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449178    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449181    4660 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449191    4660 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449194    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449198    4660 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449202    4660 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:03:20.464176 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449205    4660 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:03:20.464708 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449208    4660 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:03:20.464708 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449210    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:03:20.464708 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449213    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:03:20.464708 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449215    4660 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:03:20.464708 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449218    4660 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:03:20.464708 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449220    4660 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:03:20.464708 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449223    4660 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:03:20.464708 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449225    4660 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:03:20.464708 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449228    4660 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:03:20.464708 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449230    4660 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:03:20.464708 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449233    4660 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:03:20.464708 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449235    4660 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:03:20.464708 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449238    4660 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:03:20.464708 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449241    4660 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:03:20.464708 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.449263    4660 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 13:03:20.464708 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449312    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449317    4660 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449320    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449324    4660 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449327    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449330    4660 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449333    4660 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449336    4660 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449338    4660 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449341    4660 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449343    4660 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449346    4660 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449348    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449357    4660 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449360    4660 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449362    4660 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449364    4660 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449367    4660 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449369    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449372    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:03:20.465110 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449374    4660 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449377    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449379    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449382    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449385    4660 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449387    4660 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449390    4660 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449392    4660 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449395    4660 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449397    4660 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449399    4660 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449402    4660 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449405    4660 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449407    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449410    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449412    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449415    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449417    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449420    4660 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:03:20.465601 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449422    4660 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449424    4660 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449427    4660 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449429    4660 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449432    4660 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449434    4660 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449436    4660 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449444    4660 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449447    4660 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449450    4660 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449452    4660 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449454    4660 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449457    4660 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449459    4660 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449462    4660 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449464    4660 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449467    4660 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449469    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449471    4660 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:03:20.466060 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449474    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449477    4660 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449479    4660 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449481    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449484    4660 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449486    4660 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449489    4660 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449491    4660 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449494    4660 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449498    4660 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449501    4660 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449504    4660 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449507    4660 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449510    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449514    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449516    4660 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449519    4660 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449521    4660 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449524    4660 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449526    4660 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:03:20.466559 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449532    4660 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449540    4660 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449543    4660 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449546    4660 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449548    4660 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449551    4660 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449554    4660 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449556    4660 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449559    4660 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449562    4660 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449564    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449567    4660 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449569    4660 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449572    4660 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449576    4660 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449580    4660 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449583    4660 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449586    4660 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449589    4660 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:03:20.467029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449592    4660 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:03:20.467561 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449594    4660 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:03:20.467561 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449597    4660 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:03:20.467561 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449599    4660 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:03:20.467561 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449602    4660 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:03:20.467561 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449604    4660 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:03:20.467561 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449606    4660 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:03:20.467561 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449609    4660 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:03:20.467561 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:20.449611    4660 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:03:20.467561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.449675    4660 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 13:03:20.467561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.450708    4660 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 13:03:20.467561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.450765    4660 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 13:03:20.467561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.450961    4660 server.go:1014] "Starting client certificate rotation"
Oct 31 13:03:20.467561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.451141    4660 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 13:03:20.467561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.451178    4660 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:57:38 +0000 UTC" deadline="2028-06-26 15:11:28.272150276 +0000 UTC"
Oct 31 13:03:20.467561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.451243    4660 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14498h8m7.820911533s"
Oct 31 13:03:20.467561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.451518    4660 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 13:03:20.468281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.454050    4660 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 13:03:20.468281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.455582    4660 log.go:25] "Validated CRI v1 runtime API"
Oct 31 13:03:20.468281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.457256    4660 log.go:25] "Validated CRI v1 image API"
Oct 31 13:03:20.468281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.457966    4660 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 13:03:20.468281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.459697    4660 fs.go:135] Filesystem UUIDs: map[44866b4a-6aa7-4638-a2cf-3d07c10dbe23:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9acc9f80-b538-4aa1-b3d8-41ba6c48aa50:/dev/nvme0n1p3]
Oct 31 13:03:20.468548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.459716    4660 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ffd7d88d151c8943f6d8be4d3eedd3a685287828150bf4e48d8f6e1c49bc81e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ffd7d88d151c8943f6d8be4d3eedd3a685287828150bf4e48d8f6e1c49bc81e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b31cae7f3687a6535930f31be851f095451286b328692760cc6fc6091d3753c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b31cae7f3687a6535930f31be851f095451286b328692760cc6fc6091d3753c/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33997268971ea96102f3de9388fad7d021e710012861e1d37576d6f712a8a5d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33997268971ea96102f3de9388fad7d021e710012861e1d37576d6f712a8a5d1/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38fcb4fc076605b04085b272d1f7f926f064ed61bb8405313fbbd8fd9a304031/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38fcb4fc076605b04085b272d1f7f926f064ed61bb8405313fbbd8fd9a304031/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44b73c2b1801c7d3118e6e3a35ddc76b869c3cf3efd15d23f7fe6daf4cd5cd1f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44b73c2b1801c7d3118e6e3a35ddc76b869c3cf3efd15d23f7fe6daf4cd5cd1f/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fb8c67ec53005d8de87676cd553dbf750cda2a2224001820bc08b616b3f2301/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fb8c67ec53005d8de87676cd553dbf750cda2a2224001820bc08b616b3f2301/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57e5d116ca0671ac2013d8c117f970d0c73e267b60213885528a3d1552422885/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57e5d116ca0671ac2013d8c117f970d0c73e267b60213885528a3d1552422885/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d5ae92c54631a6c2bc8a6a035a0d9c7c471076a30ae65887af26777dc6a2f94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d5ae92c54631a6c2bc8a6a035a0d9c7c471076a30ae65887af26777dc6a2f94/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f02d4ca8a60e4f8e2b6872b04f9a1558e59163e3c22f70e90bf1c15c4969f8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f02d4ca8a60e4f8e2b6872b04f9a1558e59163e3c22f70e90bf1c15c4969f8c/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f5198b5f2308e0b0ff3e2a419abe5102d986aebb4cb3e465a7196b59a975aa4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f5198b5f2308e0b0ff3e2a419abe5102d986aebb4cb3e465a7196b59a975aa4/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/803b51df10bd848ac4f47db7333a15708b97fdf101ae7f66f1e3e2c8bbdc93f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/803b51df10bd848ac4f47db7333a15708b97fdf101ae7f66f1e3e2c8bbdc93f9/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88ca662f63a235a6f55dcd3b3d8dfd5b3de712fee6e60ccbee3a45ed68aecd59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88ca662f63a235a6f55dcd3b3d8dfd5b3de712fee6e60ccbee3a45ed68aecd59/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9253a55016ae2ece3d5b6b87601d30ec447c842a6b46d6e31ca96ac3c777f7b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9253a55016ae2ece3d5b6b87601d30ec447c842a6b46d6e31ca96ac3c777f7b7/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c9410fc6253e95927cb81986debfac06c5abf46cc78eacce0938c3c9c695500/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c9410fc6253e95927cb81986debfac06c5abf46cc78eacce0938c3c9c695500/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/adcde96d03f7d70b8276d274c287de71b7fa3d0a4f50f3c85000cbd579628aed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/adcde96d03f7d70b8276d274c287de71b7fa3d0a4f50f3c85000cbd579628aed/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae7fa7db046613ffd1b237c73aee825e86027157fec24499164ae7e3b7488b4f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae7fa7db046613ffd1b237c73aee825e86027157fec24499164ae7e3b7488b4f/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c80f48133c950dec0e233909b963cd0cc96431128ace1a9e433afccf4e6277b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c80f48133c950dec0e233909b963cd0cc96431128ace1a9e433afccf4e6277b8/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ccb24845f500d1d06d2b8d8d16f767b93a14d4481610bd3fb5ffd9302912c6b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ccb24845f500d1d06d2b8d8d16f767b93a14d4481610bd3fb5ffd9302912c6b4/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd46746273f2cb2f5eb0958dd8ca05d009bc6f0619dd403def4fb826f2baaf78/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd46746273f2cb2f5eb0958dd8ca05d009bc6f0619dd403def4fb826f2baaf78/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d39cc227edf77ab01436fdd077d8415a9a47ee6a97b912072628557ae181a5d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d39cc227edf77ab01436fdd077d8415a9a47ee6a97b912072628557ae181a5d3/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d911c55ba548c6c1bd2b4c510c219287e2c4c1402d7a679dc648bfe5e6ffd553/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d911c55ba548c6c1bd2b4c510c219287e2c4c1402d7a679dc648bfe5e6ffd553/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe6bc3dfc6c7cecac1cc8804b658445b3bdcaa3b17bfe4c626ad61f2ea0e2a7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe6bc3dfc6c7cecac1cc8804b658445b3bdcaa3b17bfe4c626ad61f2ea0e2a7e/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1984f406-7241-4b44-99c2-75fcef391d20/volumes/kubernetes.io~projected/kube-api-access-hf67n:{mountpoint:/var/lib/kubelet/pods/1984f406-7241-4b44-99c2-75fcef391d20/volumes/kubernetes.io~projected/kube-api-access-hf67n major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/209e1f84-8138-42c3-bd7a-5d1bba5457d0/volumes/kubernetes.io~projected/kube-api-access-dv7sk:{mountpoint:/var/lib/kubelet/pods/209e1f84-8138-42c3-bd7a-5d1bba5457d0/volumes/kubernetes.io~projected/kube-api-access-dv7sk major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/298ad120-4782-4b13-985f-5af7fb24d97d/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/298ad120-4782-4b13-985f-5af7fb24d97d/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/298ad120-4782-4b13-985f-5af7fb24d97d/volumes/kubernetes.io~projected/kube-api-access-gf2v9:{mountpoint:/var/lib/kubelet/pods/298ad120-4782-4b13-985f-5af7fb24d97d/volumes/kubernetes.io~projected/kube-api-access-gf2v9 major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/298ad120-4782-4b13-985f-5af7fb24d97d/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/298ad120-4782-4b13-985f-5af7fb24d97d/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/298ad120-4782-4b13-985f-5af7fb24d97d/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/298ad120-4782-4b13-985f-5af7fb24d97d/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a1a0fc2-36f2-4635-97b6-97faed580133/volumes/kubernetes.io~projected/kube-api-access-bjwq4:{mountpoint:/var/lib/kubelet/pods/2a1a0fc2-36f2-4635-97b6-97faed580133/volumes/kubernetes.io~projected/kube-api-access-bjwq4 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a1a0fc2-36f2-4635-97b6-97faed580133/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/2a1a0fc2-36f2-4635-97b6-97faed580133/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40395d51-6cc7-44b3-8cad-d4a44489e25f/volumes/kubernetes.io~projected/kube-api-access-9k967:{mountpoint:/var/lib/kubelet/pods/40395d51-6cc7-44b3-8cad-d4a44489e25f/volumes/kubernetes.io~projected/kube-api-access-9k967 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41b93592-66ac-4e0e-acf9-11271af65e89/volumes/kubernetes.io~projected/kube-api-access-nc4vh:{mountpoint:/var/lib/kubelet/pods/41b93592-66ac-4e0e-acf9-11271af65e89/volumes/kubernetes.io~projected/kube-api-access-nc4vh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4504b20c-a7d8-49e9-a572-9004fce4af1d/volumes/kubernetes.io~projected/kube-api-access-kn9c9:{mountpoint:/var/lib/kubelet/pods/4504b20c-a7d8-49e9-a572-9004fce4af1d/volumes/kubernetes.io~projected/kube-api-access-kn9c9 major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4504b20c-a7d8-49e9-a572-9004fce4af1d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4504b20c-a7d8-49e9-a572-9004fce4af1d/volumes/kubernetes.io~secret/serving-cert major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49b449ac-b419-4bd1-9896-0941450b820a/volumes/kubernetes.io~projected/kube-api-access-kh68v:{mountpoint:/var/lib/kubelet/pods/49b449ac-b419-4bd1-9896-0941450b820a/volumes/kubernetes.io~projected/kube-api-access-kh68v major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e/volumes/kubernetes.io~projected/kube-api-access-55pqf:{mountpoint:/var/lib/kubelet/pods/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e/volumes/kubernetes.io~projected/kube-api-access-55pqf major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52a6b59d-1f07-4fc3-9974-2da3a2d37676/volumes/kubernetes.io~projected/kube-api-access-r8zz6:{mountpoint:/var/lib/kubelet/pods/52a6b59d-1f07-4fc3-9974-2da3a2d37676/volumes/kubernetes.io~projected/kube-api-access-r8zz6 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58af76a6-3167-4627-bb9f-2c172665a8eb/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/58af76a6-3167-4627-bb9f-2c172665a8eb/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58af76a6-3167-4627-bb9f-2c172665a8eb/volumes/kubernetes.io~projected/kube-api-access-v649w:{mountpoint:/var/lib/kubelet/pods/58af76a6-3167-4627-bb9f-2c172665a8eb/volumes/kubernetes.io~projected/kube-api-access-v649w major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58af76a6-3167-4627-bb9f-2c172665a8eb/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/58af76a6-3167-4627-bb9f-2c172665a8eb/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7231ca35-6551-41ea-b341-46f8836425c1/volumes/kubernetes.io~projected/kube-api-access-qk2rw:{mountpoint:/var/lib/kubelet/pods/7231ca35-6551-41ea-b341-46f8836425c1/volumes/kubernetes.io~projected/kube-api-access-qk2rw major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7620273c-a4f1-4e1f-ba26-453913b2e171/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/7620273c-a4f1-4e1f-ba26-453913b2e171/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7620273c-a4f1-4e1f-ba26-453913b2e171/volumes/kubernetes.io~projected/kube-api-access-mnzhh:{mountpoint:/var/lib/kubelet/pods/7620273c-a4f1-4e1f-ba26-453913b2e171/volumes/kubernetes.io~projected/kube-api-access-mnzhh major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7620273c-a4f1-4e1f-ba26-453913b2e171/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/7620273c-a4f1-4e1f-ba26-453913b2e171/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7620273c-a4f1-4e1f-ba26-453913b2e171/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/7620273c-a4f1-4e1f-ba26-453913b2e171/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7810b5de-3907-4a6e-b11e-0c631c148438/volumes/kubernetes.io~projected/kube-api-access-f4x2q:{mountpoint:/var/lib/kubelet/pods/7810b5de-3907-4a6e-b11e-0c631c148438/volumes/kubernetes.io~projected/kube-api-access-f4x2q major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c98a759-eb02-4d1b-a9f8-f1cb77e1590a/volumes/kubernetes.io~projected/kube-api-access-5dmxw:{mountpoint:/var/lib/kubelet/pods/8c98a759-eb02-4d1b-a9f8-f1cb77e1590a/volumes/kubernetes.io~projected/kube-api-access-5dmxw major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91204e03-5b1b-4084-a699-ac7b59db4be5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/91204e03-5b1b-4084-a699-ac7b59db4be5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a096aa5d-63cc-41a6-ad78-bc64d2d33e94/volumes/kubernetes.io~projected/kube-api-access-x5pvx:{mountpoint:/var/lib/kubelet/pods/a096aa5d-63cc-41a6-ad78-bc64d2d33e94/volumes/kubernetes.io~projected/kube-api-access-x5pvx major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aab0721f-f591-4a46-b56e-27848f31642f/volumes/kubernetes.io~projected/kube-api-access-jpzp2:{mountpoint:/var/lib/kubelet/pods/aab0721f-f591-4a46-b56e-27848f31642f/volumes/kubernetes.io~projected/kube-api-access-jpzp2 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad027e2c-c49c-4c9b-8253-12acc36a1c9c/volumes/kubernetes.io~projected/kube-api-access-n7cm8:{mountpoint:/var/lib/kubelet/pods/ad027e2c-c49c-4c9b-8253-12acc36a1c9c/volumes/kubernetes.io~projected/kube-api-access-n7cm8 major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3bc5c9b-54fb-477f-8c98-9884b1b2e159/volumes/kubernetes.io~projected/kube-api-access-7j64s:{mountpoint:/var/lib/kubelet/pods/b3bc5c9b-54fb-477f-8c98-9884b1b2e159/volumes/kubernetes.io~projected/kube-api-access-7j64s major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3bc5c9b-54fb-477f-8c98-9884b1b2e159/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b3bc5c9b-54fb-477f-8c98-9884b1b2e159/volumes/kubernetes.io~secret/serving-cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360/volumes/kubernetes.io~projected/kube-api-access-z7wbz:{mountpoint:/var/lib/kubelet/pods/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360/volumes/kubernetes.io~projected/kube-api-access-z7wbz major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd718975-d0cb-4a21-8383-c29164ae4f31/volumes/kubernetes.io~projected/kube-api-access-4hbtp:{mountpoint:/var/lib/kubelet/pods/bd718975-d0cb-4a21-8383-c29164ae4f31/volumes/kubernetes.io~projected/kube-api-access-4hbtp major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd718975-d0cb-4a21-8383-c29164ae4f31/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/bd718975-d0cb-4a21-8383-c29164ae4f31/volumes/kubernetes.io~secret/cert major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdce0f75-852b-49b6-9da3-f662bb82bf94/volumes/kubernetes.io~projected/kube-api-access-t6f7q:{mountpoint:/var/lib/kubelet/pods/bdce0f75-852b-49b6-9da3-f662bb82bf94/volumes/kubernetes.io~projected/kube-api-access-t6f7q major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8d1062-0bb5-46f2-98bf-e4d52689294c/volumes/kubernetes.io~projected/kube-api-access-hqcgc:{mountpoint:/var/lib/kubelet/pods/bf8d1062-0bb5-46f2-98bf-e4d52689294c/volumes/kubernetes.io~projected/kube-api-access-hqcgc major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0b71943-a5a0-4227-9b85-161248021630/volumes/kubernetes.io~projected/kube-api-access-qwkhj:{mountpoint:/var/lib/kubelet/pods/c0b71943-a5a0-4227-9b85-161248021630/volumes/kubernetes.io~projected/kube-api-access-qwkhj major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c95a45a3-7757-4843-9161-6cdbf64385cd/volumes/kubernetes.io~projected/kube-api-access-ccsk6:{mountpoint:/var/lib/kubelet/pods/c95a45a3-7757-4843-9161-6cdbf64385cd/volumes/kubernetes.io~projected/kube-api-access-ccsk6 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c95a45a3-7757-4843-9161-6cdbf64385cd/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/c95a45a3-7757-4843-9161-6cdbf64385cd/volumes/kubernetes.io~secret/default-certificate major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c95a45a3-7757-4843-9161-6cdbf64385cd/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/c95a45a3-7757-4843-9161-6cdbf64385cd/volumes/kubernetes.io~secret/stats-auth major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf350772-aff8-4051-820a-dd114e01e441/volumes/kubernetes.io~projected/kube-api-access-tnpmh:{mountpoint:/var/lib/kubelet/pods/cf350772-aff8-4051-820a-dd114e01e441/volumes/kubernetes.io~projected/kube-api-access-tnpmh major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc5cb602-ea83-47e3-8966-b832277cb06c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/dc5cb602-ea83-47e3-8966-b832277cb06c/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a/volumes/kubernetes.io~projected/kube-api-access-rn96p:{mountpoint:/var/lib/kubelet/pods/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a/volumes/kubernetes.io~projected/kube-api-access-rn96p major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a/volumes/kubernetes.io~secret/serving-cert major:0 minor:223 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5190da5a6c68522b88d28d4e7731ba90290aa739de82799fc247c21139a7c7f1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/bc5396916d833c435a9953b1ee5f369253e6e0b445682bea163f39e7c2ad5d1f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1e81512a9370f46f77bde4ab4e0f2ffbc1b730772d7b736f0c8a31596f1095f7/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/a736c5e170f533958af0a22150b1fa8a256b2292d989f76b74a44f50c836c035/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/4c06379da7228ed445b4b5c5d7b4e68ae062286d0da61727fbcd82f9a05200d5/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/21de13a5fbfaa4deae4032ef1d43416dcab77b3ad12984ec446b5d8f494872f9/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/350207f8c2ab81b891b7a65584c743a446c3d36c5f2a1f9bd74495ac1be85017/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b4b3f8b50e77c4703469bdc8b6850bea2a3c29b069343641ff2c37b5d0c6e99f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/688d69d730fae9ddbbb9b6cc47744221876ebe1cc2a07653b6a11d6a61f1ca25/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/2195742b427eba323d7fec44b1fc9c571ac4c1be525ff708efd2d75640a214ca/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/1675201e800ae78a276eab7c5211712f9f9fd1907dd20c1f8b0c4f0f2e86b6fa/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/7271eb696cd5d162d8f04b7c0f7e620d3259a91b6d9acef56bcb0cd06cf15295/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/5b37bacf3c46c34fbbf490ecccd0b5fa1a5991e15cc055de5ee236fe84f9d127/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/f5a192977b803a65131548427198982da911e76dc57ca0cde21ec44ef0ca5372/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/0280018ad8a123f0a86f20b230dd370b7e8a978afa1c3470289f0fad7f8ec591/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/3869eb2816162b88ab262a5ba6273d2148b586cfd3dfdcc27d21d737a2556a1c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/694aca2cf3cf5eddf192796a902aa394830335ad096a671c7473e7ae2252035d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/ec1515370da648fef1492407106f5eb2a2eff7986eed4b2b4aad5c74984b8650/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d9ea568033475a8cd6228e28b7a58a8ef478a0a660310527dfc19b4afb0eef68/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b5d7d6f22a02ad557d7721892f049c95ff23132cbcd498cacde4ffe6533950c6/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/68285b5a27a712e519c13ba415a2dca10f6112a521711dd03a0e7223744799ef/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/43d5d66c0912d6c397b547eae4596e88ef48946e4f821181a2f6239fb87ccbdf/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/5611ab9e8993aa3a754317606bb220bd8e8926777415419607ca968b3175fa4f/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/674d4e9699dc5149842b99dc662891d40dedb9fafc06851905c6ebfdc848433e/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/60329f3af2f4c9649f7789acc630add748f8db1c4574d6a99cbd200698ad320d/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/19f6aa35e7f4c2540e1222264eb32f88fe93b9282281a0b6d15c68ae2f8ca249/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/94566203adc27d681a12d5126f26dcb8e237b531e154f870bf229941b5855b4e/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/53dc841a5e4804c4e5bc022074ac2bb438e34c1f21669e8703cceaa9b6b1b768/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/08d6f4d805ced8114c66d34ec0d0f9f716ef8220eddcbf901036237b3d686b83/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/1f51f03de0f29b8dc0e82412551efe2128521353a8c6d62eab7b28529fa26e64/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/f197b0b61b13ef8bacf9b4e3004593fb5ba580343c5414b3bf66fe08412762ea/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/5a8965740f28081cb554e7340a38e9f0683023e89ddfa6548681cce60e8f2a11/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/ed968b834489cc77e074009d52f88818ccf9f5f1c35735f3ebdeb017eb3812e7/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/790819f89925c6d8b1f4b07e1853088222954124df05f59696852c0f661cc132/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/d8e6db869a4723fee739ea9bf6c22f2b391791e90d9b1ccaebf5769440e0a4ec/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/e0a5ecebc440d6a5119dece4e7559567e0cc5f8af3705a204f60c4db5e701e58/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/6004c50a39bad73059f9e49a993a45bafd84147e96cdf31226bdf3715e77c1ad/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/22871cf0374b3351e132fe688eb70b0f0912f29d43c7b814219ab2b3666169e3/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/a02ee815ea8c9b36e27d0744e563a070d06b28388e0743b2c7a0efe50a0caf5d/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/23513d4dd59a3b02dfb664266a94c3af95db38cc6ab33af1fc17c21cfe50faf6/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/c88c51afe9201498669f0444289c68158b0e738391c495ae98aed7611d37f8e9/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/4c03d6c887c8ef39571fe902f02807f9905bf3236e78fe3dd9690d22ab620afa/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/5ad13025a8baf9cfe3feacf8e6b79e52ed8211810104ad1bb2b20d74c3d574eb/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/09a5fc7493f9387b33017526c4fb3812624bac46eac8193aab0984b35fba692e/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/89525f64bd80e4546f6ee181800f83f80f92553020db79731f3c73aa680b0418/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/7acc628d7bb2a49c3e61380235ad4f460f9449a132799d72ad8f64f54089c021/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/30e1466af2d471303651b241927c5d3aaa7e7754a3e5491dc68ec1ce8f51adb6/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/ab03be1c81af8c1d0bad70c2be6eaacfc8a65faab4cddded56aca8c2903261d1/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/9ce8a473138cd382620f026e467ee3ed86a17fdb284c4d0df22a9457dac7cb0c/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/d188c81bb9ae7b915dd472f36b7d4b56d0d94b65b9fe262cc999e9895a5346a9/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/400cfe79fd87fa397f0adfec03a33de369314680df5bcdb26b66f37d607dd375/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/13c6c6052b8a8d41ce4ee81c9b4d2007f0c27a8b886339745dc9e50674024444/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8132a8047f3a9b6bb1b1faecd2554893be6020f4b342ca85670216df217ea50a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4f7724b7bb5ffb7381593dedddd2139c3e2baeeb4259c51be6dcbb401abf6fe9/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.466992    4660 manager.go:217] Machine: {Timestamp:2026-10-31 13:03:20.466511434 +0000 UTC m=+0.085170589 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099830 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aa0ca0e97aa1660ea71ffba8a411a SystemUUID:ec2aa0ca-0e97-aa16-60ea-71ffba8a411a BootID:edd0cc28-4c91-4fbc-a087-a1b0bbf3f59b Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/58af76a6-3167-4627-bb9f-2c172665a8eb/volumes/kubernetes.io~projected/kube-api-access-v649w DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c95a45a3-7757-4843-9161-6cdbf64385cd/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c9410fc6253e95927cb81986debfac06c5abf46cc78eacce0938c3c9c695500/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/58af76a6-3167-4627-bb9f-2c172665a8eb/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c80f48133c950dec0e233909b963cd0cc96431128ace1a9e433afccf4e6277b8/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58af76a6-3167-4627-bb9f-2c172665a8eb/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/298ad120-4782-4b13-985f-5af7fb24d97d/volumes/kubernetes.io~projected/kube-api-access-gf2v9 DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bdce0f75-852b-49b6-9da3-f662bb82bf94/volumes/kubernetes.io~projected/kube-api-access-t6f7q DeviceMajor:0 DeviceMinor:238 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1984f406-7241-4b44-99c2-75fcef391d20/volumes/kubernetes.io~projected/kube-api-access-hf67n DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360/volumes/kubernetes.io~projected/kube-api-access-z7wbz DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c95a45a3-7757-4843-9161-6cdbf64385cd/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57e5d116ca0671ac2013d8c117f970d0c73e267b60213885528a3d1552422885/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9253a55016ae2ece3d5b6b87601d30ec447c842a6b46d6e31ca96ac3c777f7b7/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe6bc3dfc6c7cecac1cc8804b658445b3bdcaa3b17bfe4c626ad61f2ea0e2a7e/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b3bc5c9b-54fb-477f-8c98-9884b1b2e159/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/803b51df10bd848ac4f47db7333a15708b97fdf101ae7f66f1e3e2c8bbdc93f9/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e/volumes/kubernetes.io~projected/kube-api-access-55pqf DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40395d51-6cc7-44b3-8cad-d4a44489e25f/volumes/kubernetes.io~projected/kube-api-access-9k967 DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae7fa7db046613ffd1b237c73aee825e86027157fec24499164ae7e3b7488b4f/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/91204e03-5b1b-4084-a699-ac7b59db4be5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7810b5de-3907-4a6e-b11e-0c631c148438/volumes/kubernetes.io~projected/kube-api-access-f4x2q DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/298ad120-4782-4b13-985f-5af7fb24d97d/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a1a0fc2-36f2-4635-97b6-97faed580133/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7620273c-a4f1-4e1f-ba26-453913b2e171/volumes/kubernetes.io~projected/kube-api-access-mnzhh DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c98a759-eb02-4d1b-a9f8-f1cb77e1590a/volumes/kubernetes.io~projected/kube-api-access-5dmxw DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4504b20c-a7d8-49e9-a572-9004fce4af1d/volumes/kubernetes.io~projected/kube-api-access-kn9c9 DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/41b93592-66ac-4e0e-acf9-11271af65e89/volumes/kubernetes.io~projected/kube-api-access-nc4vh DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/298ad120-4782-4b13-985f-5af7fb24d97d/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a/volumes/kubernetes.io~projected/kube-api-access-rn96p DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/49b449ac-b419-4bd1-9896-0941450b820a/volumes/kubernetes.io~projected/kube-api-access-kh68v DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7620273c-a4f1-4e1f-ba26-453913b2e171/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/298ad120-4782-4b13-985f-5af7fb24d97d/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c95a45a3-7757-4843-9161-6cdbf64385cd/volumes/kubernetes.io~projected/kube-api-access-ccsk6 DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f02d4ca8a60e4f8e2b6872b04f9a1558e59163e3c22f70e90bf1c15c4969f8c/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ccb24845f500d1d06d2b8d8d16f767b93a14d4481610bd3fb5ffd9302912c6b4/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d5ae92c54631a6c2bc8a6a035a0d9c7c471076a30ae65887af26777dc6a2f94/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d911c55ba548c6c1bd2b4c510c219287e2c4c1402d7a679dc648bfe5e6ffd553/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/209e1f84-8138-42c3-bd7a-5d1bba5457d0/volumes/kubernetes.io~projected/kube-api-access-dv7sk DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38fcb4fc076605b04085b272d1f7f926f064ed61bb8405313fbbd8fd9a304031/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c0b71943-a5a0-4227-9b85-161248021630/volumes/kubernetes.io~projected/kube-api-access-qwkhj DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4504b20c-a7d8-49e9-a572-9004fce4af1d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33997268971ea96102f3de9388fad7d021e710012861e1d37576d6f712a8a5d1/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b31cae7f3687a6535930f31be851f095451286b328692760cc6fc6091d3753c/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7620273c-a4f1-4e1f-ba26-453913b2e171/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7231ca35-6551-41ea-b341-46f8836425c1/volumes/kubernetes.io~projected/kube-api-access-qk2rw DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a096aa5d-63cc-41a6-ad78-bc64d2d33e94/volumes/kubernetes.io~projected/kube-api-access-x5pvx DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd718975-d0cb-4a21-8383-c29164ae4f31/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a1a0fc2-36f2-4635-97b6-97faed580133/volumes/kubernetes.io~projected/kube-api-access-bjwq4 DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ffd7d88d151c8943f6d8be4d3eedd3a685287828150bf4e48d8f6e1c49bc81e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d39cc227edf77ab01436fdd077d8415a9a47ee6a97b912072628557ae181a5d3/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad027e2c-c49c-4c9b-8253-12acc36a1c9c/volumes/kubernetes.io~projected/kube-api-access-n7cm8 DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88ca662f63a235a6f55dcd3b3d8dfd5b3de712fee6e60ccbee3a45ed68aecd59/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd46746273f2cb2f5eb0958dd8ca05d009bc6f0619dd403def4fb826f2baaf78/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf350772-aff8-4051-820a-dd114e01e441/volumes/kubernetes.io~projected/kube-api-access-tnpmh DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd718975-d0cb-4a21-8383-c29164ae4f31/volumes/kubernetes.io~projected/kube-api-access-4hbtp DeviceMajor:0 DeviceMinor:239 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52a6b59d-1f07-4fc3-9974-2da3a2d37676/volumes/kubernetes.io~projected/kube-api-access-r8zz6 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b3bc5c9b-54fb-477f-8c98-9884b1b2e159/volumes/kubernetes.io~projected/kube-api-access-7j64s DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f5198b5f2308e0b0ff3e2a419abe5102d986aebb4cb3e465a7196b59a975aa4/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44b73c2b1801c7d3118e6e3a35ddc76b869c3cf3efd15d23f7fe6daf4cd5cd1f/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aab0721f-f591-4a46-b56e-27848f31642f/volumes/kubernetes.io~projected/kube-api-access-jpzp2 DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7620273c-a4f1-4e1f-ba26-453913b2e171/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8d1062-0bb5-46f2-98bf-e4d52689294c/volumes/kubernetes.io~projected/kube-api-access-hqcgc DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc5cb602-ea83-47e3-8966-b832277cb06c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/adcde96d03f7d70b8276d274c287de71b7fa3d0a4f50f3c85000cbd579628aed/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fb8c67ec53005d8de87676cd553dbf750cda2a2224001820bc08b616b3f2301/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:38fcb4fc076605b MacAddress:a2:c0:cf:2a:3d:2d Speed:10000 Mtu:8901} {Name:44b73c2b1801c7d MacAddress:0e:42:9d:f9:8a:b1 Speed:10000 Mtu:8901} {Name:4fb8c67ec53005d MacAddress:f2:88:f6:e7:2d:94 Speed:10000 Mtu:8901} {Name:57e5d116ca0671a MacAddress:a2:0f:34:70:f7:57 Speed:10000 Mtu:8901} {Name:5d5ae92c54631a6 MacAddress:3a:1d:1a:08:c9:69 Speed:10000 Mtu:8901} {Name:5f02d4ca8a60e4f MacAddress:ce:ae:10:a7:54:27 Speed:10000 Mtu:8901} {Name:803b51df10bd848 MacAddress:9a:7b:87:38:20:58 Speed:10000 Mtu:8901} {Name:88ca662f63a235a MacAddress:5a:c3:98:31:d8:40 Speed:10000 Mtu:8901} {Name:9c9410fc6253e95 MacAddress:76:ae:1a:8c:c2:20 Speed:10000 Mtu:8901} {Name:ae7fa7db046613f MacAddress:6e:0c:42:85:0e:96 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cd:fc:f3:35 Speed:0 Mtu:9001} {Name:br-int MacAddress:ee:9d:74:ae:29:42 Speed:0 Mtu:8901} {Name:d39cc227edf77ab MacAddress:ba:56:8e:a3:21:36 Speed:10000 Mtu:8901} {Name:d911c55ba548c6c MacAddress:32:28:9d:db:38:64 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cd:fc:f3:35 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:9a:cc:a3:f3:f9:a5 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2e:86:a3:cf:81:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.467333    4660 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.467372    4660 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.467809    4660 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.467837    4660 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-77.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.468048    4660 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.468061    4660 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.468079    4660 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.468106    4660 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.468198    4660 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.468480    4660 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.468629    4660 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.468673    4660 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.468686    4660 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.468699    4660 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.468711    4660 kubelet.go:405] "Adding apiserver pod source"
Oct 31 13:03:20.468891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.468723    4660 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 13:03:20.471732 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.470886    4660 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 13:03:20.471732 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.470938    4660 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 13:03:20.472316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.472192    4660 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 13:03:20.474419 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.474402    4660 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 13:03:20.474419 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.474417    4660 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 13:03:20.474571 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.474526    4660 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 13:03:20.474758 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.474742    4660 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 13:03:20.474819 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.474783    4660 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 13:03:20.474905 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.474894    4660 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 13:03:20.474959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.474913    4660 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 13:03:20.474959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.474924    4660 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 13:03:20.474959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.474934    4660 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 13:03:20.474959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.474943    4660 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 13:03:20.474959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.474952    4660 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 13:03:20.475183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.474962    4660 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 13:03:20.475183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.474972    4660 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 13:03:20.475183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.474984    4660 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 13:03:20.475183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.474994    4660 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 13:03:20.475183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.475013    4660 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 13:03:20.475183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.475026    4660 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 13:03:20.475183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.475058    4660 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 13:03:20.475183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.475068    4660 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 13:03:20.478529 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.478514    4660 server.go:1263] "Started kubelet"
Oct 31 13:03:20.478641 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.478585    4660 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 13:03:20.479075 ip-10-0-137-77 systemd[1]: Started Kubernetes Kubelet.
Oct 31 13:03:20.479615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.479001    4660 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 13:03:20.479615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.479131    4660 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 13:03:20.481343 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.480351    4660 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 13:03:20.481343 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.480442    4660 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 13:03:20.482558 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.482538    4660 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 13:03:20.482645 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.482571    4660 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 13:03:20.483062 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.483043    4660 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 13:03:20.483062 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.483063    4660 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 13:03:20.483190 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.483176    4660 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 13:03:20.483190 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:20.483045    4660 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-77.us-east-2.compute.internal\" not found"
Oct 31 13:03:20.484536 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.484516    4660 factory.go:55] Registering systemd factory
Oct 31 13:03:20.484615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.484540    4660 factory.go:223] Registration of the systemd container factory successfully
Oct 31 13:03:20.486199 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.486170    4660 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:57:40 +0000 UTC" deadline="2028-05-17 13:46:24.781040464 +0000 UTC"
Oct 31 13:03:20.486278 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.486201    4660 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13536h43m4.294843827s"
Oct 31 13:03:20.488320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488267    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4504b20c-a7d8-49e9-a572-9004fce4af1d" volumeName="kubernetes.io/configmap/4504b20c-a7d8-49e9-a572-9004fce4af1d-trusted-ca" seLinuxMountContext=""
Oct 31 13:03:20.488430 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488411    4660 factory.go:153] Registering CRI-O factory
Oct 31 13:03:20.488430 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488431    4660 factory.go:223] Registration of the crio container factory successfully
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488506    4660 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488534    4660 factory.go:103] Registering Raw factory
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488429    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="298ad120-4782-4b13-985f-5af7fb24d97d" volumeName="kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-registry-certificates" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488550    4660 manager.go:1199] Started watching for new ooms in manager
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488555    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41b93592-66ac-4e0e-acf9-11271af65e89" volumeName="kubernetes.io/projected/41b93592-66ac-4e0e-acf9-11271af65e89-kube-api-access-nc4vh" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488582    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a6b59d-1f07-4fc3-9974-2da3a2d37676" volumeName="kubernetes.io/projected/52a6b59d-1f07-4fc3-9974-2da3a2d37676-kube-api-access-r8zz6" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488597    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7620273c-a4f1-4e1f-ba26-453913b2e171" volumeName="kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-installation-pull-secrets" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488611    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a096aa5d-63cc-41a6-ad78-bc64d2d33e94" volumeName="kubernetes.io/projected/a096aa5d-63cc-41a6-ad78-bc64d2d33e94-kube-api-access-x5pvx" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488630    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad027e2c-c49c-4c9b-8253-12acc36a1c9c" volumeName="kubernetes.io/projected/ad027e2c-c49c-4c9b-8253-12acc36a1c9c-kube-api-access-n7cm8" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488643    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8d1062-0bb5-46f2-98bf-e4d52689294c" volumeName="kubernetes.io/configmap/bf8d1062-0bb5-46f2-98bf-e4d52689294c-telemetry-config" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488666    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8d1062-0bb5-46f2-98bf-e4d52689294c" volumeName="kubernetes.io/projected/bf8d1062-0bb5-46f2-98bf-e4d52689294c-kube-api-access-hqcgc" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488681    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0b71943-a5a0-4227-9b85-161248021630" volumeName="kubernetes.io/projected/c0b71943-a5a0-4227-9b85-161248021630-kube-api-access-qwkhj" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488698    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c95a45a3-7757-4843-9161-6cdbf64385cd" volumeName="kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-stats-auth" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488712    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e" volumeName="kubernetes.io/empty-dir/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-tmp" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488725    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7620273c-a4f1-4e1f-ba26-453913b2e171" volumeName="kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488741    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91204e03-5b1b-4084-a699-ac7b59db4be5" volumeName="kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488758    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3bc5c9b-54fb-477f-8c98-9884b1b2e159" volumeName="kubernetes.io/secret/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-serving-cert" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488858    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5ed0a8a-fd41-44ec-8c25-fa7421d3b360" volumeName="kubernetes.io/projected/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-kube-api-access-z7wbz" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488870    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0b71943-a5a0-4227-9b85-161248021630" volumeName="kubernetes.io/configmap/c0b71943-a5a0-4227-9b85-161248021630-iptables-alerter-script" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488886    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1a0fc2-36f2-4635-97b6-97faed580133" volumeName="kubernetes.io/configmap/2a1a0fc2-36f2-4635-97b6-97faed580133-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488897    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="209e1f84-8138-42c3-bd7a-5d1bba5457d0" volumeName="kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488909    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1a0fc2-36f2-4635-97b6-97faed580133" volumeName="kubernetes.io/configmap/2a1a0fc2-36f2-4635-97b6-97faed580133-service-ca-bundle" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488924    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58af76a6-3167-4627-bb9f-2c172665a8eb" volumeName="kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-ovnkube-config" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488937    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7231ca35-6551-41ea-b341-46f8836425c1" volumeName="kubernetes.io/projected/7231ca35-6551-41ea-b341-46f8836425c1-kube-api-access-qk2rw" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488955    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7620273c-a4f1-4e1f-ba26-453913b2e171" volumeName="kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-kube-api-access-mnzhh" seLinuxMountContext=""
Oct 31 13:03:20.488967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488968    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1984f406-7241-4b44-99c2-75fcef391d20" volumeName="kubernetes.io/projected/1984f406-7241-4b44-99c2-75fcef391d20-kube-api-access-hf67n" seLinuxMountContext=""
Oct 31 13:03:20.490019 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488995    4660 manager.go:319] Starting recovery of all containers
Oct 31 13:03:20.490019 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.488995    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="298ad120-4782-4b13-985f-5af7fb24d97d" volumeName="kubernetes.io/empty-dir/298ad120-4782-4b13-985f-5af7fb24d97d-ca-trust-extracted" seLinuxMountContext=""
Oct 31 13:03:20.490019 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.489545    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a6b59d-1f07-4fc3-9974-2da3a2d37676" volumeName="kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.491682    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73811006-7011-4339-800d-5f1b8a9bbf88" volumeName="kubernetes.io/configmap/73811006-7011-4339-800d-5f1b8a9bbf88-nginx-conf" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.496952    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7620273c-a4f1-4e1f-ba26-453913b2e171" volumeName="kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-bound-sa-token" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.496971    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd718975-d0cb-4a21-8383-c29164ae4f31" volumeName="kubernetes.io/secret/bd718975-d0cb-4a21-8383-c29164ae4f31-cert" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.496984    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdce0f75-852b-49b6-9da3-f662bb82bf94" volumeName="kubernetes.io/projected/bdce0f75-852b-49b6-9da3-f662bb82bf94-kube-api-access-t6f7q" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.496997    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c95a45a3-7757-4843-9161-6cdbf64385cd" volumeName="kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-default-certificate" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497009    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1a0fc2-36f2-4635-97b6-97faed580133" volumeName="kubernetes.io/empty-dir/2a1a0fc2-36f2-4635-97b6-97faed580133-snapshots" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497019    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7620273c-a4f1-4e1f-ba26-453913b2e171" volumeName="kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-certificates" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497031    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7620273c-a4f1-4e1f-ba26-453913b2e171" volumeName="kubernetes.io/empty-dir/7620273c-a4f1-4e1f-ba26-453913b2e171-ca-trust-extracted" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497042    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aab0721f-f591-4a46-b56e-27848f31642f" volumeName="kubernetes.io/projected/aab0721f-f591-4a46-b56e-27848f31642f-kube-api-access-jpzp2" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497055    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd718975-d0cb-4a21-8383-c29164ae4f31" volumeName="kubernetes.io/projected/bd718975-d0cb-4a21-8383-c29164ae4f31-kube-api-access-4hbtp" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497067    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf350772-aff8-4051-820a-dd114e01e441" volumeName="kubernetes.io/projected/cf350772-aff8-4051-820a-dd114e01e441-kube-api-access-tnpmh" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497079    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a" volumeName="kubernetes.io/configmap/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-config" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497091    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="298ad120-4782-4b13-985f-5af7fb24d97d" volumeName="kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-trusted-ca" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497102    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="298ad120-4782-4b13-985f-5af7fb24d97d" volumeName="kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-kube-api-access-gf2v9" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497113    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4504b20c-a7d8-49e9-a572-9004fce4af1d" volumeName="kubernetes.io/configmap/4504b20c-a7d8-49e9-a572-9004fce4af1d-config" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497125    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c98a759-eb02-4d1b-a9f8-f1cb77e1590a" volumeName="kubernetes.io/projected/8c98a759-eb02-4d1b-a9f8-f1cb77e1590a-kube-api-access-5dmxw" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497138    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aab0721f-f591-4a46-b56e-27848f31642f" volumeName="kubernetes.io/empty-dir/aab0721f-f591-4a46-b56e-27848f31642f-data-volume" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497151    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdce0f75-852b-49b6-9da3-f662bb82bf94" volumeName="kubernetes.io/empty-dir/bdce0f75-852b-49b6-9da3-f662bb82bf94-tmp-dir" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497165    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40395d51-6cc7-44b3-8cad-d4a44489e25f" volumeName="kubernetes.io/projected/40395d51-6cc7-44b3-8cad-d4a44489e25f-kube-api-access-9k967" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497177    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e" volumeName="kubernetes.io/empty-dir/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-tuned" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497190    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c95a45a3-7757-4843-9161-6cdbf64385cd" volumeName="kubernetes.io/projected/c95a45a3-7757-4843-9161-6cdbf64385cd-kube-api-access-ccsk6" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497204    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="298ad120-4782-4b13-985f-5af7fb24d97d" volumeName="kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-bound-sa-token" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497216    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4504b20c-a7d8-49e9-a572-9004fce4af1d" volumeName="kubernetes.io/secret/4504b20c-a7d8-49e9-a572-9004fce4af1d-serving-cert" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497229    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a6b59d-1f07-4fc3-9974-2da3a2d37676" volumeName="kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497240    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7810b5de-3907-4a6e-b11e-0c631c148438" volumeName="kubernetes.io/configmap/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-config" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497251    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3bc5c9b-54fb-477f-8c98-9884b1b2e159" volumeName="kubernetes.io/projected/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-kube-api-access-7j64s" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497282    4660 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497286    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc5cb602-ea83-47e3-8966-b832277cb06c" volumeName="kubernetes.io/configmap/dc5cb602-ea83-47e3-8966-b832277cb06c-konnectivity-ca" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497396    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a6b59d-1f07-4fc3-9974-2da3a2d37676" volumeName="kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cni-binary-copy" seLinuxMountContext=""
Oct 31 13:03:20.497395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497414    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58af76a6-3167-4627-bb9f-2c172665a8eb" volumeName="kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-env-overrides" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497429    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7231ca35-6551-41ea-b341-46f8836425c1" volumeName="kubernetes.io/empty-dir/7231ca35-6551-41ea-b341-46f8836425c1-tmp-dir" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497454    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49b449ac-b419-4bd1-9896-0941450b820a" volumeName="kubernetes.io/projected/49b449ac-b419-4bd1-9896-0941450b820a-kube-api-access-kh68v" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497467    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1a0fc2-36f2-4635-97b6-97faed580133" volumeName="kubernetes.io/projected/2a1a0fc2-36f2-4635-97b6-97faed580133-kube-api-access-bjwq4" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497480    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1a0fc2-36f2-4635-97b6-97faed580133" volumeName="kubernetes.io/secret/2a1a0fc2-36f2-4635-97b6-97faed580133-serving-cert" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497494    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7810b5de-3907-4a6e-b11e-0c631c148438" volumeName="kubernetes.io/projected/7810b5de-3907-4a6e-b11e-0c631c148438-kube-api-access-f4x2q" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497507    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a" volumeName="kubernetes.io/projected/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-kube-api-access-rn96p" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497519    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a" volumeName="kubernetes.io/secret/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-serving-cert" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497533    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1a0fc2-36f2-4635-97b6-97faed580133" volumeName="kubernetes.io/empty-dir/2a1a0fc2-36f2-4635-97b6-97faed580133-tmp" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497564    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4504b20c-a7d8-49e9-a572-9004fce4af1d" volumeName="kubernetes.io/projected/4504b20c-a7d8-49e9-a572-9004fce4af1d-kube-api-access-kn9c9" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497576    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e" volumeName="kubernetes.io/projected/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-kube-api-access-55pqf" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497590    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7620273c-a4f1-4e1f-ba26-453913b2e171" volumeName="kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-trusted-ca" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497602    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="298ad120-4782-4b13-985f-5af7fb24d97d" volumeName="kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-installation-pull-secrets" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497615    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58af76a6-3167-4627-bb9f-2c172665a8eb" volumeName="kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497628    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58af76a6-3167-4627-bb9f-2c172665a8eb" volumeName="kubernetes.io/secret/58af76a6-3167-4627-bb9f-2c172665a8eb-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497640    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3bc5c9b-54fb-477f-8c98-9884b1b2e159" volumeName="kubernetes.io/configmap/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-config" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497654    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5ed0a8a-fd41-44ec-8c25-fa7421d3b360" volumeName="kubernetes.io/configmap/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-cni-binary-copy" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497668    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdce0f75-852b-49b6-9da3-f662bb82bf94" volumeName="kubernetes.io/configmap/bdce0f75-852b-49b6-9da3-f662bb82bf94-config-volume" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497680    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc5cb602-ea83-47e3-8966-b832277cb06c" volumeName="kubernetes.io/secret/dc5cb602-ea83-47e3-8966-b832277cb06c-agent-certs" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497694    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="298ad120-4782-4b13-985f-5af7fb24d97d" volumeName="kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497707    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41b93592-66ac-4e0e-acf9-11271af65e89" volumeName="kubernetes.io/configmap/41b93592-66ac-4e0e-acf9-11271af65e89-serviceca" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497726    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58af76a6-3167-4627-bb9f-2c172665a8eb" volumeName="kubernetes.io/projected/58af76a6-3167-4627-bb9f-2c172665a8eb-kube-api-access-v649w" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497738    4660 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5ed0a8a-fd41-44ec-8c25-fa7421d3b360" volumeName="kubernetes.io/configmap/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-daemon-config" seLinuxMountContext=""
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497751    4660 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497757    4660 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.497984    4660 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:03:20.498799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.498654    4660 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:03:20.501422 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.501407    4660 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 13:03:20.510332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.510310    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/1.log"
Oct 31 13:03:20.510410 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.510350    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/1.log"
Oct 31 13:03:20.517469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.517426    4660 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 13:03:20.519336 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.518918    4660 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 13:03:20.519336 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.518945    4660 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 13:03:20.519336 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.518967    4660 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 13:03:20.519336 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:20.519009    4660 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 13:03:20.524588 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.523384    4660 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:03:20.537342 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.537320    4660 generic.go:356] "Generic (PLEG): container finished" podID="52a6b59d-1f07-4fc3-9974-2da3a2d37676" containerID="ae6467ec5cd1cdf06b4d3531b0e0927e98daa86012876cf0b623bd1020c701f0" exitCode=0
Oct 31 13:03:20.537342 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.537339    4660 generic.go:356] "Generic (PLEG): container finished" podID="52a6b59d-1f07-4fc3-9974-2da3a2d37676" containerID="16cfe8274ab2227ebab4171f6859458891420030a6f73c5a822e940d75ed13b9" exitCode=0
Oct 31 13:03:20.537455 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.537346    4660 generic.go:356] "Generic (PLEG): container finished" podID="52a6b59d-1f07-4fc3-9974-2da3a2d37676" containerID="d994739fee67995b5042e1413a6c04c7df5934e5296e3d0cdbf56e1322c2d3cd" exitCode=0
Oct 31 13:03:20.537455 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.537351    4660 generic.go:356] "Generic (PLEG): container finished" podID="52a6b59d-1f07-4fc3-9974-2da3a2d37676" containerID="c43e8b1db395c800016096817723f4ff51fc4c45c58b7f8a127e5b19e5c7f839" exitCode=0
Oct 31 13:03:20.537455 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.537356    4660 generic.go:356] "Generic (PLEG): container finished" podID="52a6b59d-1f07-4fc3-9974-2da3a2d37676" containerID="17815dfdacf0a85a55813284ba62dd50b172b863c097aecb06ddfe025cf39e54" exitCode=0
Oct 31 13:03:20.537455 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.537361    4660 generic.go:356] "Generic (PLEG): container finished" podID="52a6b59d-1f07-4fc3-9974-2da3a2d37676" containerID="f6842a9b919f5b76312a37ac743212119f326f9c7a0ab3e8acdb9b8f2b9a77ba" exitCode=0
Oct 31 13:03:20.560670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.560592    4660 generic.go:356] "Generic (PLEG): container finished" podID="71ed2bb6e8396f1c7d68918710e8a4ed" containerID="436dce848fbda3bf0615f40c0b2e9191dd87cf2a0018481296db4b516b85c770" exitCode=0
Oct 31 13:03:20.567379 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.567354    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/1.log"
Oct 31 13:03:20.567471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.567392    4660 generic.go:356] "Generic (PLEG): container finished" podID="4504b20c-a7d8-49e9-a572-9004fce4af1d" containerID="3d8b55e30b935d5d939f5ab6056e8c94bd638608bc9e562793f30465208bd095" exitCode=255
Oct 31 13:03:20.587178 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.587038    4660 manager.go:324] Recovery completed
Oct 31 13:03:20.606590 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.606564    4660 cpu_manager.go:226] "Starting" policy="none"
Oct 31 13:03:20.606590 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.606585    4660 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 13:03:20.606704 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.606603    4660 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 13:03:20.606749 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.606723    4660 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 13:03:20.606749 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.606734    4660 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 13:03:20.606803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.606751    4660 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 13:03:20.606803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.606757    4660 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 13:03:20.606803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.606762    4660 policy_none.go:50] "Start"
Oct 31 13:03:20.606803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.606770    4660 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 13:03:20.606803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.606779    4660 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 13:03:20.606929 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.606899    4660 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 13:03:20.606929 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.606905    4660 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 13:03:20.606929 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.606910    4660 policy_none.go:44] "Start"
Oct 31 13:03:20.611656 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.611631    4660 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 13:03:20.614289 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.614271    4660 manager.go:341] "Starting Device Plugin manager"
Oct 31 13:03:20.614879 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:20.614312    4660 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 13:03:20.614930 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.614883    4660 server.go:87] "Starting device plugin registration server"
Oct 31 13:03:20.615107 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.615096    4660 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 13:03:20.615144 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.615111    4660 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 13:03:20.615220 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.615198    4660 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 13:03:20.615356 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.615328    4660 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 13:03:20.615430 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.615383    4660 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 13:03:20.615430 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.615391    4660 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 13:03:20.615515 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.615434    4660 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T13:03:20.615354145Z","UUID":"8bf64d59-5cf5-4560-a628-52d88ac69b52","Handler":null,"Name":"","Endpoint":""}
Oct 31 13:03:20.616567 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:20.616528    4660 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 13:03:20.616919 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.616901    4660 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 13:03:20.616999 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.616927    4660 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 13:03:20.619115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.619062    4660 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"]
Oct 31 13:03:20.619893 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.619852    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal" event={"ID":"e08a66f57394b6ed04cdde42605f9314","Type":"ContainerStarted","Data":"2d4eff280414325dbddceb438588253f4c644b0c9baf4a084d0e1f9e2f88471b"}
Oct 31 13:03:20.619972 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.619897    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal" event={"ID":"e08a66f57394b6ed04cdde42605f9314","Type":"ContainerStarted","Data":"33997268971ea96102f3de9388fad7d021e710012861e1d37576d6f712a8a5d1"}
Oct 31 13:03:20.620025 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.619973    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal" event={"ID":"71ed2bb6e8396f1c7d68918710e8a4ed","Type":"ContainerStarted","Data":"ed3857d1c329f17abb2b228f69ec835e938ff347c855aab46901756b2d80bee0"}
Oct 31 13:03:20.620025 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.619986    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal" event={"ID":"71ed2bb6e8396f1c7d68918710e8a4ed","Type":"ContainerDied","Data":"436dce848fbda3bf0615f40c0b2e9191dd87cf2a0018481296db4b516b85c770"}
Oct 31 13:03:20.620025 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.620001    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal" event={"ID":"71ed2bb6e8396f1c7d68918710e8a4ed","Type":"ContainerStarted","Data":"ccb24845f500d1d06d2b8d8d16f767b93a14d4481610bd3fb5ffd9302912c6b4"}
Oct 31 13:03:20.620025 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.620019    4660 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:20.620199 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.620113    4660 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:20.633732 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.633701    4660 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 13:03:20.633812 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:20.633756    4660 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:20.634625 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.634613    4660 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 13:03:20.634666 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:20.634639    4660 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:20.703368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.703344    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e08a66f57394b6ed04cdde42605f9314-config\") pod \"kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"e08a66f57394b6ed04cdde42605f9314\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:20.703474 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.703371    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71ed2bb6e8396f1c7d68918710e8a4ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"71ed2bb6e8396f1c7d68918710e8a4ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:20.703474 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.703389    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71ed2bb6e8396f1c7d68918710e8a4ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"71ed2bb6e8396f1c7d68918710e8a4ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:20.715447 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.715424    4660 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:03:20.716781 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.716765    4660 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:03:20.716842 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.716793    4660 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:03:20.716842 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.716803    4660 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-77.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:03:20.716842 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.716842    4660 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:20.730082 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.730065    4660 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:20.730172 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.730127    4660 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:20.804442 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.804418    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e08a66f57394b6ed04cdde42605f9314-config\") pod \"kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"e08a66f57394b6ed04cdde42605f9314\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:20.804442 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.804432    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e08a66f57394b6ed04cdde42605f9314-config\") pod \"kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"e08a66f57394b6ed04cdde42605f9314\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:20.804582 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.804462    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71ed2bb6e8396f1c7d68918710e8a4ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"71ed2bb6e8396f1c7d68918710e8a4ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:20.804582 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.804483    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71ed2bb6e8396f1c7d68918710e8a4ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"71ed2bb6e8396f1c7d68918710e8a4ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:20.804582 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.804521    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71ed2bb6e8396f1c7d68918710e8a4ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"71ed2bb6e8396f1c7d68918710e8a4ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:20.804582 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:20.804553    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71ed2bb6e8396f1c7d68918710e8a4ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal\" (UID: \"71ed2bb6e8396f1c7d68918710e8a4ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:21.469976 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.469942    4660 apiserver.go:52] "Watching apiserver"
Oct 31 13:03:21.481206 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.481179    4660 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 13:03:21.481856 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.481833    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76fcd8cbb5-tmzzx","openshift-multus/network-metrics-daemon-pphvg","openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft","openshift-ingress-canary/ingress-canary-l42ds","kube-system/kas-connection-checker-685895c7c9-dswrj","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns","openshift-network-diagnostics/network-check-target-tkc46","openshift-image-registry/node-ca-lgcv8","openshift-insights/insights-operator-7c4986d449-cpk9z","openshift-console-operator/console-operator-7cd56b8dbc-7v9w9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal","kube-system/kas-connection-checker-685895c7c9-wvdhx","openshift-multus/multus-dskj5","openshift-network-operator/iptables-alerter-2d5zt","kube-system/kas-connection-checker-685895c7c9-sqxqp","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w7wlc","openshift-cluster-node-tuning-operator/tuned-dfnfj","openshift-dns/dns-default-pztwr","openshift-monitoring/cluster-monitoring-operator-884576c57-44698","kube-system/global-pull-secret-syncer-hrj9c","openshift-image-registry/image-registry-694db45b8d-szb8q","openshift-network-console/networking-console-plugin-86c478bb88-swrc7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb","kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s","openshift-network-diagnostics/network-check-source-6f78d59f99-lcmgp","openshift-ingress/router-default-848dcf9575-7trg4","openshift-ovn-kubernetes/ovnkube-node-7nv8r","kube-system/konnectivity-agent-mttz6","openshift-dns/node-resolver-gpl2l","openshift-insights/insights-runtime-extractor-9hjjp","openshift-multus/multus-additional-cni-plugins-x27jz"]
Oct 31 13:03:21.483188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.483153    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:21.483576 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.483553    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.483732 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.483714    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:21.484182 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.484164    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:21.487077 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.487061    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:21.489784 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.489760    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 13:03:21.489870 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.489862    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 13:03:21.490004 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.489990    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pfgb9\""
Oct 31 13:03:21.490106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.490085    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.490258 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.490243    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.490358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.490330    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 13:03:21.490542 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.490527    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.490666 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.490645    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.490749 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.490735    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.490793 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.490528    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 13:03:21.490793 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.490785    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 13:03:21.490882 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.490828    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 13:03:21.490882 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.490843    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 13:03:21.491082 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.491067    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 13:03:21.491130 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.491067    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 13:03:21.491193 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.491179    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 13:03:21.492108 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.492086    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 13:03:21.492192 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.492178    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 13:03:21.492524 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.492504    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.492616 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.492540    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.492671 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.492622    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.492749 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.492734    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.493778 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.493750    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 13:03:21.493992 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.493977    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 13:03:21.494054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.493992    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.494124 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.494110    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rw847\""
Oct 31 13:03:21.494173 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.494118    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.494173 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.494139    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.494173 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.494149    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 13:03:21.494350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.494172    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.494350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.494190    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 13:03:21.494350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.494143    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 13:03:21.494350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.494204    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.494350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.494210    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 13:03:21.494350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.494175    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zrqm9\""
Oct 31 13:03:21.494350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.494246    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8dvth\""
Oct 31 13:03:21.494350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.494256    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.494350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.494350    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.494732 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.494556    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vq7zb\""
Oct 31 13:03:21.494793 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.494776    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.494861 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.494845    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ftcbc\""
Oct 31 13:03:21.495149 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.495128    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.495215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.495176    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 13:03:21.495590 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.495572    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rlfbc\""
Oct 31 13:03:21.495679 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.495609    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7x7tk\""
Oct 31 13:03:21.496787 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.496768    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:21.497981 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.497965    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 13:03:21.498156 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.498137    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 13:03:21.498238 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.498161    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.498412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.498397    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.498737 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.498722    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 13:03:21.499484 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.499465    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:21.499965 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.499943    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.500172 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.500155    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:21.500898 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.500884    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.501666 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.501647    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 13:03:21.501738 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.501652    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-np6pg\""
Oct 31 13:03:21.501784 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.501768    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s8mtw\""
Oct 31 13:03:21.501848 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.501829    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 13:03:21.501889 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.501855    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.501889 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.501857    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 13:03:21.502179 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.502163    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.502425 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.502403    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 13:03:21.502490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.502424    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.502490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.502445    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 13:03:21.502490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.502477    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 13:03:21.502580 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.502489    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 13:03:21.502580 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.502496    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.502580 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.502506    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.502580 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.502530    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.502738 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.502722    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.502820 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.502805    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.502958 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.502938    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 13:03:21.503012 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.502945    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.503012 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.502963    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r2wwf\""
Oct 31 13:03:21.503733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.503713    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-glfvs\""
Oct 31 13:03:21.503845 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.503826    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 13:03:21.503944 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.503925    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5rqjw\""
Oct 31 13:03:21.504047 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.503749    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-h72qs\""
Oct 31 13:03:21.504115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.503733    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 13:03:21.504499 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.504472    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wr5k7\""
Oct 31 13:03:21.504584 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.504565    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 13:03:21.504691 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.504676    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tmkp5\""
Oct 31 13:03:21.504825 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.504806    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 13:03:21.504876 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.504724    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lxhpt\""
Oct 31 13:03:21.504964 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.504944    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 13:03:21.505035 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.505012    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j8xvf\""
Oct 31 13:03:21.505172 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.505154    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-65v8h\""
Oct 31 13:03:21.505460 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.505443    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 13:03:21.505574 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.505555    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wdnw7\""
Oct 31 13:03:21.505658 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.505643    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.505859 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.505843    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 13:03:21.505961 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.505945    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-s9d6g\""
Oct 31 13:03:21.506024 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.505968    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.506087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.506057    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.506087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.506081    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 13:03:21.506244 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.506226    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkxw2\""
Oct 31 13:03:21.506315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.506273    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 13:03:21.506380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.506335    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 13:03:21.506380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.506341    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.506463 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.506380    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 13:03:21.506506 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.506468    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 13:03:21.506571 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.506508    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 13:03:21.506626 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.506610    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 13:03:21.506749 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.506734    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 13:03:21.506818 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.506779    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 13:03:21.506875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.506833    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tmfz7\""
Oct 31 13:03:21.506940 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.506913    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qbkfg\""
Oct 31 13:03:21.507044 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507025    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dmxw\" (UniqueName: \"kubernetes.io/projected/8c98a759-eb02-4d1b-a9f8-f1cb77e1590a-kube-api-access-5dmxw\") pod \"kas-connection-checker-685895c7c9-sqxqp\" (UID: \"8c98a759-eb02-4d1b-a9f8-f1cb77e1590a\") " pod="kube-system/kas-connection-checker-685895c7c9-sqxqp"
Oct 31 13:03:21.507098 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507069    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:21.507148 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507096    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7cm8\" (UniqueName: \"kubernetes.io/projected/ad027e2c-c49c-4c9b-8253-12acc36a1c9c-kube-api-access-n7cm8\") pod \"kas-connection-checker-685895c7c9-wvdhx\" (UID: \"ad027e2c-c49c-4c9b-8253-12acc36a1c9c\") " pod="kube-system/kas-connection-checker-685895c7c9-wvdhx"
Oct 31 13:03:21.507148 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507127    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-default-certificate\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:21.507251 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507161    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:21.507251 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507194    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-image-registry-private-configuration\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.507251 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507224    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-installation-pull-secrets\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.507446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507257    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7620273c-a4f1-4e1f-ba26-453913b2e171-ca-trust-extracted\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.507446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507307    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-stats-auth\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:21.507446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507339    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccsk6\" (UniqueName: \"kubernetes.io/projected/c95a45a3-7757-4843-9161-6cdbf64385cd-kube-api-access-ccsk6\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:21.507446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507368    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.507446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507398    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-certificates\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.507446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507423    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-bound-sa-token\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.507710 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507452    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzhh\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-kube-api-access-mnzhh\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.507710 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507482    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k967\" (UniqueName: \"kubernetes.io/projected/40395d51-6cc7-44b3-8cad-d4a44489e25f-kube-api-access-9k967\") pod \"kas-connection-checker-685895c7c9-dswrj\" (UID: \"40395d51-6cc7-44b3-8cad-d4a44489e25f\") " pod="kube-system/kas-connection-checker-685895c7c9-dswrj"
Oct 31 13:03:21.507710 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507512    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-trusted-ca\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.507872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.507767    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7620273c-a4f1-4e1f-ba26-453913b2e171-ca-trust-extracted\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.508326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.508265    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-certificates\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.508437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.508415    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-image-registry-private-configuration\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.508502 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.508464    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-default-certificate\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:21.508560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.508508    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-stats-auth\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:21.509031 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.509012    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-installation-pull-secrets\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.509100 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.509060    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-trusted-ca\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.518923 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.518897    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7cm8\" (UniqueName: \"kubernetes.io/projected/ad027e2c-c49c-4c9b-8253-12acc36a1c9c-kube-api-access-n7cm8\") pod \"kas-connection-checker-685895c7c9-wvdhx\" (UID: \"ad027e2c-c49c-4c9b-8253-12acc36a1c9c\") " pod="kube-system/kas-connection-checker-685895c7c9-wvdhx"
Oct 31 13:03:21.519162 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.519143    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzhh\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-kube-api-access-mnzhh\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.519361 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.519346    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccsk6\" (UniqueName: \"kubernetes.io/projected/c95a45a3-7757-4843-9161-6cdbf64385cd-kube-api-access-ccsk6\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:21.519463 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.519445    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dmxw\" (UniqueName: \"kubernetes.io/projected/8c98a759-eb02-4d1b-a9f8-f1cb77e1590a-kube-api-access-5dmxw\") pod \"kas-connection-checker-685895c7c9-sqxqp\" (UID: \"8c98a759-eb02-4d1b-a9f8-f1cb77e1590a\") " pod="kube-system/kas-connection-checker-685895c7c9-sqxqp"
Oct 31 13:03:21.521179 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.521161    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k967\" (UniqueName: \"kubernetes.io/projected/40395d51-6cc7-44b3-8cad-d4a44489e25f-kube-api-access-9k967\") pod \"kas-connection-checker-685895c7c9-dswrj\" (UID: \"40395d51-6cc7-44b3-8cad-d4a44489e25f\") " pod="kube-system/kas-connection-checker-685895c7c9-dswrj"
Oct 31 13:03:21.521336 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.521309    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-bound-sa-token\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.570358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.570341    4660 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:21.570493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.570480    4660 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:21.582851 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.582835    4660 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 13:03:21.582936 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.582872    4660 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:21.583849 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.583834    4660 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 13:03:21.583905 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.583868    4660 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-77.us-east-2.compute.internal"
Oct 31 13:03:21.585237 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.585223    4660 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 13:03:21.608227 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608207    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-kubernetes\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.608325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608229    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-var-lib-kubelet\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.608325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608248    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7sk\" (UniqueName: \"kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk\") pod \"network-check-target-tkc46\" (UID: \"209e1f84-8138-42c3-bd7a-5d1bba5457d0\") " pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:03:21.608325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608265    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-socket-dir-parent\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.608481 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608340    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-var-lib-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.608481 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608376    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-k8s-cni-cncf-io\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.608481 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608404    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-netns\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.608481 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608437    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4x2q\" (UniqueName: \"kubernetes.io/projected/7810b5de-3907-4a6e-b11e-0c631c148438-kube-api-access-f4x2q\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:21.608481 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608462    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-lib-modules\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.608701 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608547    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a1a0fc2-36f2-4635-97b6-97faed580133-snapshots\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:21.608701 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608600    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91204e03-5b1b-4084-a699-ac7b59db4be5-kubelet-config\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:21.608701 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608654    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-cni-bin\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.608701 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608687    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-conf-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.608860 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608722    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-multus-certs\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.608860 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608746    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn9c9\" (UniqueName: \"kubernetes.io/projected/4504b20c-a7d8-49e9-a572-9004fce4af1d-kube-api-access-kn9c9\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:21.608860 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608766    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4504b20c-a7d8-49e9-a572-9004fce4af1d-trusted-ca\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:21.608860 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608813    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a1a0fc2-36f2-4635-97b6-97faed580133-snapshots\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:21.609006 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608868    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:21.609006 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608928    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73811006-7011-4339-800d-5f1b8a9bbf88-nginx-conf\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:21.609006 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608960    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cnibin\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.609006 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.608991    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.609186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609036    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s\" (UID: \"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:21.609186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609064    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysconfig\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.609186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609088    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-kubelet\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.609186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609114    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-env-overrides\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.609186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609144    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/73811006-7011-4339-800d-5f1b8a9bbf88-nginx-conf\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:21.609186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609164    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v649w\" (UniqueName: \"kubernetes.io/projected/58af76a6-3167-4627-bb9f-2c172665a8eb-kube-api-access-v649w\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.609186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609160    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4504b20c-a7d8-49e9-a572-9004fce4af1d-trusted-ca\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:21.609670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609216    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-etc-kubernetes\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.609670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609254    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.609670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609262    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s\" (UID: \"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:21.609670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609288    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-trusted-ca\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.609670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609335    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c0b71943-a5a0-4227-9b85-161248021630-iptables-alerter-script\") pod \"iptables-alerter-2d5zt\" (UID: \"c0b71943-a5a0-4227-9b85-161248021630\") " pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:03:21.609670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609359    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-cni-binary-copy\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.609670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609398    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-env-overrides\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.609670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609491    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.609670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609524    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-modprobe-d\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.609670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609560    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-image-registry-private-configuration\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.609670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609591    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf67n\" (UniqueName: \"kubernetes.io/projected/1984f406-7241-4b44-99c2-75fcef391d20-kube-api-access-hf67n\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.609670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609612    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c0b71943-a5a0-4227-9b85-161248021630-iptables-alerter-script\") pod \"iptables-alerter-2d5zt\" (UID: \"c0b71943-a5a0-4227-9b85-161248021630\") " pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:03:21.609670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609615    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-cni-binary-copy\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.609670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609604    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.609670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609639    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnpmh\" (UniqueName: \"kubernetes.io/projected/cf350772-aff8-4051-820a-dd114e01e441-kube-api-access-tnpmh\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609742    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-ovn\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609839    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-image-registry-private-configuration\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609850    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh68v\" (UniqueName: \"kubernetes.io/projected/49b449ac-b419-4bd1-9896-0941450b820a-kube-api-access-kh68v\") pod \"network-check-source-6f78d59f99-lcmgp\" (UID: \"49b449ac-b419-4bd1-9896-0941450b820a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lcmgp"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609883    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-bound-sa-token\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.609980    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bdce0f75-852b-49b6-9da3-f662bb82bf94-config-volume\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610029    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a1a0fc2-36f2-4635-97b6-97faed580133-serving-cert\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610062    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-cni-netd\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610086    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-cnibin\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610102    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-trusted-ca\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610114    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41b93592-66ac-4e0e-acf9-11271af65e89-serviceca\") pod \"node-ca-lgcv8\" (UID: \"41b93592-66ac-4e0e-acf9-11271af65e89\") " pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610139    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bdce0f75-852b-49b6-9da3-f662bb82bf94-config-volume\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610140    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-systemd-units\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610216    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-ovnkube-config\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610245    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd718975-d0cb-4a21-8383-c29164ae4f31-cert\") pod \"ingress-canary-l42ds\" (UID: \"bd718975-d0cb-4a21-8383-c29164ae4f31\") " pod="openshift-ingress-canary/ingress-canary-l42ds"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610244    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a1a0fc2-36f2-4635-97b6-97faed580133-serving-cert\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610288    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-socket-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610359    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610367    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41b93592-66ac-4e0e-acf9-11271af65e89-serviceca\") pod \"node-ca-lgcv8\" (UID: \"41b93592-66ac-4e0e-acf9-11271af65e89\") " pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610381    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-cni-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610418    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/298ad120-4782-4b13-985f-5af7fb24d97d-ca-trust-extracted\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610435    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-installation-pull-secrets\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610451    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-hostroot\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610465    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4504b20c-a7d8-49e9-a572-9004fce4af1d-serving-cert\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610478    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd718975-d0cb-4a21-8383-c29164ae4f31-cert\") pod \"ingress-canary-l42ds\" (UID: \"bd718975-d0cb-4a21-8383-c29164ae4f31\") " pod="openshift-ingress-canary/ingress-canary-l42ds"
Oct 31 13:03:21.610469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610484    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610492    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-ovnkube-config\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610507    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-device-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610521    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/298ad120-4782-4b13-985f-5af7fb24d97d-ca-trust-extracted\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610595    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4504b20c-a7d8-49e9-a572-9004fce4af1d-serving-cert\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610620    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1a0fc2-36f2-4635-97b6-97faed580133-service-ca-bundle\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610653    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7231ca35-6551-41ea-b341-46f8836425c1-hosts-file\") pod \"node-resolver-gpl2l\" (UID: \"7231ca35-6551-41ea-b341-46f8836425c1\") " pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610703    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk2rw\" (UniqueName: \"kubernetes.io/projected/7231ca35-6551-41ea-b341-46f8836425c1-kube-api-access-qk2rw\") pod \"node-resolver-gpl2l\" (UID: \"7231ca35-6551-41ea-b341-46f8836425c1\") " pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610747    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-installation-pull-secrets\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610775    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-systemd\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610785    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1a0fc2-36f2-4635-97b6-97faed580133-service-ca-bundle\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610810    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aab0721f-f591-4a46-b56e-27848f31642f-crio-socket\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610843    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-systemd\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610885    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-etc-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610921    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-log-socket\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610953    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.610995    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/58af76a6-3167-4627-bb9f-2c172665a8eb-ovn-node-metrics-cert\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611022    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611048    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611072    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysctl-d\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611099    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2v9\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-kube-api-access-gf2v9\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611139    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.611156    4660 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611194    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/58af76a6-3167-4627-bb9f-2c172665a8eb-ovn-node-metrics-cert\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611194    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwkhj\" (UniqueName: \"kubernetes.io/projected/c0b71943-a5a0-4227-9b85-161248021630-kube-api-access-qwkhj\") pod \"iptables-alerter-2d5zt\" (UID: \"c0b71943-a5a0-4227-9b85-161248021630\") " pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611263    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.611286    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:22.11126479 +0000 UTC m=+1.729923954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : secret "router-metrics-certs-default" not found
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611329    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-node-log\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611358    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-cni-multus\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611384    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-system-cni-dir\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611415    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8zz6\" (UniqueName: \"kubernetes.io/projected/52a6b59d-1f07-4fc3-9974-2da3a2d37676-kube-api-access-r8zz6\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611444    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-slash\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611474    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-cni-bin\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611496    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j64s\" (UniqueName: \"kubernetes.io/projected/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-kube-api-access-7j64s\") pod \"service-ca-operator-544d749c8c-4hrft\" (UID: \"b3bc5c9b-54fb-477f-8c98-9884b1b2e159\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611511    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a1a0fc2-36f2-4635-97b6-97faed580133-tmp\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611526    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1a0fc2-36f2-4635-97b6-97faed580133-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611567    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611596    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-run-netns\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611622    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aab0721f-f591-4a46-b56e-27848f31642f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:21.611639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611655    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-system-cni-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611702    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a1a0fc2-36f2-4635-97b6-97faed580133-tmp\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611748    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7231ca35-6551-41ea-b341-46f8836425c1-tmp-dir\") pod \"node-resolver-gpl2l\" (UID: \"7231ca35-6551-41ea-b341-46f8836425c1\") " pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611782    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqcgc\" (UniqueName: \"kubernetes.io/projected/bf8d1062-0bb5-46f2-98bf-e4d52689294c-kube-api-access-hqcgc\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611794    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7231ca35-6551-41ea-b341-46f8836425c1-tmp-dir\") pod \"node-resolver-gpl2l\" (UID: \"7231ca35-6551-41ea-b341-46f8836425c1\") " pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611840    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7wbz\" (UniqueName: \"kubernetes.io/projected/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-kube-api-access-z7wbz\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611867    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41b93592-66ac-4e0e-acf9-11271af65e89-host\") pod \"node-ca-lgcv8\" (UID: \"41b93592-66ac-4e0e-acf9-11271af65e89\") " pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611892    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1a0fc2-36f2-4635-97b6-97faed580133-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611940    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611965    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-tuned\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611982    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55pqf\" (UniqueName: \"kubernetes.io/projected/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-kube-api-access-55pqf\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.611998    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc5cb602-ea83-47e3-8966-b832277cb06c-agent-certs\") pod \"konnectivity-agent-mttz6\" (UID: \"dc5cb602-ea83-47e3-8966-b832277cb06c\") " pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612013    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-registry-certificates\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.612034    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:22.112022655 +0000 UTC m=+1.730681815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : configmap references non-existent config key: service-ca.crt
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612048    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5pvx\" (UniqueName: \"kubernetes.io/projected/a096aa5d-63cc-41a6-ad78-bc64d2d33e94-kube-api-access-x5pvx\") pod \"volume-data-source-validator-58c9f5f46d-w7wlc\" (UID: \"a096aa5d-63cc-41a6-ad78-bc64d2d33e94\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w7wlc"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612066    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612078    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-tuned\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612081    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-registration-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612131    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-etc-selinux\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612146    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-registry-certificates\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612233    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-sys\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612262    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc5cb602-ea83-47e3-8966-b832277cb06c-agent-certs\") pod \"konnectivity-agent-mttz6\" (UID: \"dc5cb602-ea83-47e3-8966-b832277cb06c\") " pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612285    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc5cb602-ea83-47e3-8966-b832277cb06c-konnectivity-ca\") pod \"konnectivity-agent-mttz6\" (UID: \"dc5cb602-ea83-47e3-8966-b832277cb06c\") " pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612335    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91204e03-5b1b-4084-a699-ac7b59db4be5-dbus\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612360    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-config\") pod \"service-ca-operator-544d749c8c-4hrft\" (UID: \"b3bc5c9b-54fb-477f-8c98-9884b1b2e159\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612378    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-sys-fs\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612396    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612409    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc5cb602-ea83-47e3-8966-b832277cb06c-konnectivity-ca\") pod \"konnectivity-agent-mttz6\" (UID: \"dc5cb602-ea83-47e3-8966-b832277cb06c\") " pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612420    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpzp2\" (UniqueName: \"kubernetes.io/projected/aab0721f-f591-4a46-b56e-27848f31642f-kube-api-access-jpzp2\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612451    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hbtp\" (UniqueName: \"kubernetes.io/projected/bd718975-d0cb-4a21-8383-c29164ae4f31-kube-api-access-4hbtp\") pod \"ingress-canary-l42ds\" (UID: \"bd718975-d0cb-4a21-8383-c29164ae4f31\") " pod="openshift-ingress-canary/ingress-canary-l42ds"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612501    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-serving-cert\") pod \"service-ca-operator-544d749c8c-4hrft\" (UID: \"b3bc5c9b-54fb-477f-8c98-9884b1b2e159\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612556    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-config\") pod \"service-ca-operator-544d749c8c-4hrft\" (UID: \"b3bc5c9b-54fb-477f-8c98-9884b1b2e159\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612601    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612655    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4504b20c-a7d8-49e9-a572-9004fce4af1d-config\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612661    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91204e03-5b1b-4084-a699-ac7b59db4be5-original-pull-secret\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612696    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-serving-cert\") pod \"service-ca-operator-544d749c8c-4hrft\" (UID: \"b3bc5c9b-54fb-477f-8c98-9884b1b2e159\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612704    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjwq4\" (UniqueName: \"kubernetes.io/projected/2a1a0fc2-36f2-4635-97b6-97faed580133-kube-api-access-bjwq4\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612728    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc4vh\" (UniqueName: \"kubernetes.io/projected/41b93592-66ac-4e0e-acf9-11271af65e89-kube-api-access-nc4vh\") pod \"node-ca-lgcv8\" (UID: \"41b93592-66ac-4e0e-acf9-11271af65e89\") " pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612744    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bf8d1062-0bb5-46f2-98bf-e4d52689294c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612786    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4504b20c-a7d8-49e9-a572-9004fce4af1d-config\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612881    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-daemon-config\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612898    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-host\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612913    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aab0721f-f591-4a46-b56e-27848f31642f-data-volume\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612931    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612952    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysctl-conf\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612966    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-run\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.612998    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bf8d1062-0bb5-46f2-98bf-e4d52689294c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613005    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aab0721f-f591-4a46-b56e-27848f31642f-data-volume\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613020    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-daemon-config\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.613032    4660 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.613043    4660 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76fcd8cbb5-tmzzx: secret "image-registry-tls" not found
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613067    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.613099    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls podName:7620273c-a4f1-4e1f-ba26-453913b2e171 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:22.113083137 +0000 UTC m=+1.731742293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls") pod "image-registry-76fcd8cbb5-tmzzx" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171") : secret "image-registry-tls" not found
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613115    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6f7q\" (UniqueName: \"kubernetes.io/projected/bdce0f75-852b-49b6-9da3-f662bb82bf94-kube-api-access-t6f7q\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613136    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-kubelet\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613152    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-tmp\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613170    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-ovnkube-script-lib\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613193    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0b71943-a5a0-4227-9b85-161248021630-host-slash\") pod \"iptables-alerter-2d5zt\" (UID: \"c0b71943-a5a0-4227-9b85-161248021630\") " pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613239    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s\" (UID: \"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613246    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-tmp\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613310    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn96p\" (UniqueName: \"kubernetes.io/projected/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-kube-api-access-rn96p\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s\" (UID: \"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613343    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cni-binary-copy\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613367    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613388    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdce0f75-852b-49b6-9da3-f662bb82bf94-tmp-dir\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613413    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613453    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/58af76a6-3167-4627-bb9f-2c172665a8eb-ovnkube-script-lib\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.613433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613453    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s\" (UID: \"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:21.615309 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613511    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cni-binary-copy\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.615309 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613540    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdce0f75-852b-49b6-9da3-f662bb82bf94-tmp-dir\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:21.615309 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.613580    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.618849 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.618830    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv7sk\" (UniqueName: \"kubernetes.io/projected/209e1f84-8138-42c3-bd7a-5d1bba5457d0-kube-api-access-dv7sk\") pod \"network-check-target-tkc46\" (UID: \"209e1f84-8138-42c3-bd7a-5d1bba5457d0\") " pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:03:21.619189 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.619173    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4x2q\" (UniqueName: \"kubernetes.io/projected/7810b5de-3907-4a6e-b11e-0c631c148438-kube-api-access-f4x2q\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:21.619799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.619784    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v649w\" (UniqueName: \"kubernetes.io/projected/58af76a6-3167-4627-bb9f-2c172665a8eb-kube-api-access-v649w\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.619946 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.619928    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn9c9\" (UniqueName: \"kubernetes.io/projected/4504b20c-a7d8-49e9-a572-9004fce4af1d-kube-api-access-kn9c9\") pod \"console-operator-7cd56b8dbc-7v9w9\" (UID: \"4504b20c-a7d8-49e9-a572-9004fce4af1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:21.620487 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.620471    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh68v\" (UniqueName: \"kubernetes.io/projected/49b449ac-b419-4bd1-9896-0941450b820a-kube-api-access-kh68v\") pod \"network-check-source-6f78d59f99-lcmgp\" (UID: \"49b449ac-b419-4bd1-9896-0941450b820a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lcmgp"
Oct 31 13:03:21.620856 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.620841    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnpmh\" (UniqueName: \"kubernetes.io/projected/cf350772-aff8-4051-820a-dd114e01e441-kube-api-access-tnpmh\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:21.625908 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.625880    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hbtp\" (UniqueName: \"kubernetes.io/projected/bd718975-d0cb-4a21-8383-c29164ae4f31-kube-api-access-4hbtp\") pod \"ingress-canary-l42ds\" (UID: \"bd718975-d0cb-4a21-8383-c29164ae4f31\") " pod="openshift-ingress-canary/ingress-canary-l42ds"
Oct 31 13:03:21.626041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.626017    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-bound-sa-token\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.626160 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.626132    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf67n\" (UniqueName: \"kubernetes.io/projected/1984f406-7241-4b44-99c2-75fcef391d20-kube-api-access-hf67n\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.626261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.626240    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn96p\" (UniqueName: \"kubernetes.io/projected/f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a-kube-api-access-rn96p\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s\" (UID: \"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s"
Oct 31 13:03:21.626346 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.626332    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk2rw\" (UniqueName: \"kubernetes.io/projected/7231ca35-6551-41ea-b341-46f8836425c1-kube-api-access-qk2rw\") pod \"node-resolver-gpl2l\" (UID: \"7231ca35-6551-41ea-b341-46f8836425c1\") " pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:03:21.626456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.626441    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqcgc\" (UniqueName: \"kubernetes.io/projected/bf8d1062-0bb5-46f2-98bf-e4d52689294c-kube-api-access-hqcgc\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:21.629650 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.629630    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwkhj\" (UniqueName: \"kubernetes.io/projected/c0b71943-a5a0-4227-9b85-161248021630-kube-api-access-qwkhj\") pod \"iptables-alerter-2d5zt\" (UID: \"c0b71943-a5a0-4227-9b85-161248021630\") " pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:03:21.629930 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.629914    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpzp2\" (UniqueName: \"kubernetes.io/projected/aab0721f-f591-4a46-b56e-27848f31642f-kube-api-access-jpzp2\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:21.630493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.630475    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6f7q\" (UniqueName: \"kubernetes.io/projected/bdce0f75-852b-49b6-9da3-f662bb82bf94-kube-api-access-t6f7q\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:21.631049 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.631029    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55pqf\" (UniqueName: \"kubernetes.io/projected/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-kube-api-access-55pqf\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.631323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.631283    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc4vh\" (UniqueName: \"kubernetes.io/projected/41b93592-66ac-4e0e-acf9-11271af65e89-kube-api-access-nc4vh\") pod \"node-ca-lgcv8\" (UID: \"41b93592-66ac-4e0e-acf9-11271af65e89\") " pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:03:21.631528 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.631506    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8zz6\" (UniqueName: \"kubernetes.io/projected/52a6b59d-1f07-4fc3-9974-2da3a2d37676-kube-api-access-r8zz6\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.631703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.631685    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7wbz\" (UniqueName: \"kubernetes.io/projected/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-kube-api-access-z7wbz\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.631776 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.631760    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5pvx\" (UniqueName: \"kubernetes.io/projected/a096aa5d-63cc-41a6-ad78-bc64d2d33e94-kube-api-access-x5pvx\") pod \"volume-data-source-validator-58c9f5f46d-w7wlc\" (UID: \"a096aa5d-63cc-41a6-ad78-bc64d2d33e94\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w7wlc"
Oct 31 13:03:21.631844 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.631829    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2v9\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-kube-api-access-gf2v9\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.632989 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.632969    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjwq4\" (UniqueName: \"kubernetes.io/projected/2a1a0fc2-36f2-4635-97b6-97faed580133-kube-api-access-bjwq4\") pod \"insights-operator-7c4986d449-cpk9z\" (UID: \"2a1a0fc2-36f2-4635-97b6-97faed580133\") " pod="openshift-insights/insights-operator-7c4986d449-cpk9z"
Oct 31 13:03:21.635459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.635443    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j64s\" (UniqueName: \"kubernetes.io/projected/b3bc5c9b-54fb-477f-8c98-9884b1b2e159-kube-api-access-7j64s\") pod \"service-ca-operator-544d749c8c-4hrft\" (UID: \"b3bc5c9b-54fb-477f-8c98-9884b1b2e159\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft"
Oct 31 13:03:21.714034 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.713998    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.714034 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714033    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-modprobe-d\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.714188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714051    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-ovn\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.714188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714122    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.714188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714148    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-modprobe-d\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.714188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714159    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-cni-netd\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.714188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714164    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-ovn\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.714188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714124    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-cni-netd\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.714467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714219    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-cnibin\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.714467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714250    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-systemd-units\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.714467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714278    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-socket-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.714467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714335    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:21.714467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714371    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-cni-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.714467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714377    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-cnibin\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.714467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714342    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-systemd-units\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.714467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714397    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-hostroot\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.714467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714440    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-socket-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.714467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714446    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-cni-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.714467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714462    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-hostroot\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.714478    4660 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.714521    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls podName:bf8d1062-0bb5-46f2-98bf-e4d52689294c nodeName:}" failed. No retries permitted until 2026-10-31 13:03:22.214506653 +0000 UTC m=+1.833165821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-44698" (UID: "bf8d1062-0bb5-46f2-98bf-e4d52689294c") : secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714538    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714559    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-device-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714574    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7231ca35-6551-41ea-b341-46f8836425c1-hosts-file\") pod \"node-resolver-gpl2l\" (UID: \"7231ca35-6551-41ea-b341-46f8836425c1\") " pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714588    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-systemd\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714620    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-systemd\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714632    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7231ca35-6551-41ea-b341-46f8836425c1-hosts-file\") pod \"node-resolver-gpl2l\" (UID: \"7231ca35-6551-41ea-b341-46f8836425c1\") " pod="openshift-dns/node-resolver-gpl2l"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714631    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-device-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714651    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aab0721f-f591-4a46-b56e-27848f31642f-crio-socket\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.714679    4660 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714706    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-systemd\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714680    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-systemd\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.714720    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert podName:73811006-7011-4339-800d-5f1b8a9bbf88 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:22.214709427 +0000 UTC m=+1.833368569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-swrc7" (UID: "73811006-7011-4339-800d-5f1b8a9bbf88") : secret "networking-console-plugin-cert" not found
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714738    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aab0721f-f591-4a46-b56e-27848f31642f-crio-socket\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714746    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-etc-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714768    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-etc-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714776    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-log-socket\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714803    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714842    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysctl-d\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714857    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-log-socket\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714867    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714892    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-node-log\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714867    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.714915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714916    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-cni-multus\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714954    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-system-cni-dir\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714958    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714933    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysctl-d\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714978    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-cni-multus\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714976    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-system-cni-dir\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714957    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-node-log\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.714997    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-slash\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715051    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-cni-bin\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715061    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-slash\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715080    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715103    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-run-netns\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715126    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aab0721f-f591-4a46-b56e-27848f31642f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715148    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-run-netns\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715161    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-cni-bin\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715149    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-system-cni-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715198    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41b93592-66ac-4e0e-acf9-11271af65e89-host\") pod \"node-ca-lgcv8\" (UID: \"41b93592-66ac-4e0e-acf9-11271af65e89\") " pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715219    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-system-cni-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715229    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715163    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715254    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41b93592-66ac-4e0e-acf9-11271af65e89-host\") pod \"node-ca-lgcv8\" (UID: \"41b93592-66ac-4e0e-acf9-11271af65e89\") " pod="openshift-image-registry/node-ca-lgcv8"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715255    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-registration-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715319    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-registration-dir\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715318    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-etc-selinux\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715356    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-sys\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.715371    4660 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715376    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91204e03-5b1b-4084-a699-ac7b59db4be5-dbus\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715405    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-etc-selinux\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715405    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-sys-fs\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715455    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715457    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1984f406-7241-4b44-99c2-75fcef391d20-sys-fs\") pod \"aws-ebs-csi-driver-node-krzsb\" (UID: \"1984f406-7241-4b44-99c2-75fcef391d20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzsb"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715472    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-sys\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715496    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-host\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715504    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91204e03-5b1b-4084-a699-ac7b59db4be5-dbus\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.715527    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs podName:cf350772-aff8-4051-820a-dd114e01e441 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:22.215496878 +0000 UTC m=+1.834156032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs") pod "network-metrics-daemon-pphvg" (UID: "cf350772-aff8-4051-820a-dd114e01e441") : secret "metrics-daemon-secret" not found
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715535    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-host\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.715537    4660 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715571    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysctl-conf\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.715578    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls podName:aab0721f-f591-4a46-b56e-27848f31642f nodeName:}" failed. No retries permitted until 2026-10-31 13:03:22.215568679 +0000 UTC m=+1.834227852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hjjp" (UID: "aab0721f-f591-4a46-b56e-27848f31642f") : secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715613    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-run\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715636    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715657    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-kubelet\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715677    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aab0721f-f591-4a46-b56e-27848f31642f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715687    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysctl-conf\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715684    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0b71943-a5a0-4227-9b85-161248021630-host-slash\") pod \"iptables-alerter-2d5zt\" (UID: \"c0b71943-a5a0-4227-9b85-161248021630\") " pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715713    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0b71943-a5a0-4227-9b85-161248021630-host-slash\") pod \"iptables-alerter-2d5zt\" (UID: \"c0b71943-a5a0-4227-9b85-161248021630\") " pod="openshift-network-operator/iptables-alerter-2d5zt"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715696    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-run\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715729    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.715735    4660 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715737    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-var-lib-kubelet\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715754    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-kubernetes\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715778    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-run-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715794    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-kubernetes\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.715798    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls podName:bdce0f75-852b-49b6-9da3-f662bb82bf94 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:22.215762733 +0000 UTC m=+1.834421896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls") pod "dns-default-pztwr" (UID: "bdce0f75-852b-49b6-9da3-f662bb82bf94") : secret "dns-default-metrics-tls" not found
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715823    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-var-lib-kubelet\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715840    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-socket-dir-parent\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715865    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-var-lib-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715885    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-var-lib-kubelet\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715890    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-k8s-cni-cncf-io\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715913    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-netns\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715923    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-socket-dir-parent\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715929    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-lib-modules\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715934    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-k8s-cni-cncf-io\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715945    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-netns\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715912    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-var-lib-openvswitch\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.715947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715957    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91204e03-5b1b-4084-a699-ac7b59db4be5-kubelet-config\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.715993    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91204e03-5b1b-4084-a699-ac7b59db4be5-kubelet-config\") pod \"global-pull-secret-syncer-hrj9c\" (UID: \"91204e03-5b1b-4084-a699-ac7b59db4be5\") " pod="kube-system/global-pull-secret-syncer-hrj9c"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716002    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-cni-bin\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716022    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-conf-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716036    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-cni-bin\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716039    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-multus-certs\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716060    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-host-run-multus-certs\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716063    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716094    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-multus-conf-dir\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.716107    4660 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716126    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cnibin\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716105    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52a6b59d-1f07-4fc3-9974-2da3a2d37676-cnibin\") pod \"multus-additional-cni-plugins-x27jz\" (UID: \"52a6b59d-1f07-4fc3-9974-2da3a2d37676\") " pod="openshift-multus/multus-additional-cni-plugins-x27jz"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.716152    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls podName:7810b5de-3907-4a6e-b11e-0c631c148438 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:22.216141874 +0000 UTC m=+1.834801019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7sfns" (UID: "7810b5de-3907-4a6e-b11e-0c631c148438") : secret "samples-operator-tls" not found
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716096    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-lib-modules\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716163    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716195    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysconfig\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.716206    4660 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716211    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-kubelet\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716228    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-etc-kubernetes\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.716214    4660 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694db45b8d-szb8q: secret "image-registry-tls" not found
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716253    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/58af76a6-3167-4627-bb9f-2c172665a8eb-host-kubelet\") pod \"ovnkube-node-7nv8r\" (UID: \"58af76a6-3167-4627-bb9f-2c172665a8eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716285    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e-etc-sysconfig\") pod \"tuned-dfnfj\" (UID: \"4d9a93ef-e1e7-40a7-82ba-51d8a1cabe5e\") " pod="openshift-cluster-node-tuning-operator/tuned-dfnfj"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.716308    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5ed0a8a-fd41-44ec-8c25-fa7421d3b360-etc-kubernetes\") pod \"multus-dskj5\" (UID: \"b5ed0a8a-fd41-44ec-8c25-fa7421d3b360\") " pod="openshift-multus/multus-dskj5"
Oct 31 13:03:21.717655 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:21.716316    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls podName:298ad120-4782-4b13-985f-5af7fb24d97d nodeName:}" failed. No retries permitted until 2026-10-31 13:03:22.216288224 +0000 UTC m=+1.834947381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls") pod "image-registry-694db45b8d-szb8q" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d") : secret "image-registry-tls" not found
Oct 31 13:03:21.785659 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:21.785608    4660 scope.go:122] "RemoveContainer" containerID="3d8b55e30b935d5d939f5ab6056e8c94bd638608bc9e562793f30465208bd095"
Oct 31 13:03:22.119766 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:22.119547    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:22.120045 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.119691    4660 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:22.120045 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.119887    4660 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76fcd8cbb5-tmzzx: secret "image-registry-tls" not found
Oct 31 13:03:22.120045 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:22.119917    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:22.120045 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.119969    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls podName:7620273c-a4f1-4e1f-ba26-453913b2e171 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:23.119947401 +0000 UTC m=+2.738606558 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls") pod "image-registry-76fcd8cbb5-tmzzx" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171") : secret "image-registry-tls" not found
Oct 31 13:03:22.120045 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.120010    4660 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 13:03:22.120045 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:22.120022    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:22.120287 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.120063    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:23.120039997 +0000 UTC m=+2.738699143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : secret "router-metrics-certs-default" not found
Oct 31 13:03:22.120287 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.120121    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:23.120108995 +0000 UTC m=+2.738768155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : configmap references non-existent config key: service-ca.crt
Oct 31 13:03:22.221243 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:22.221213    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:22.221405 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:22.221251    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:22.221405 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:22.221290    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:22.221405 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:22.221358    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:22.221405 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.221383    4660 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 13:03:22.221405 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.221387    4660 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:22.221667 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.221392    4660 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 13:03:22.221667 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:22.221385    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:22.221667 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.221436    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs podName:cf350772-aff8-4051-820a-dd114e01e441 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:23.221419507 +0000 UTC m=+2.840078656 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs") pod "network-metrics-daemon-pphvg" (UID: "cf350772-aff8-4051-820a-dd114e01e441") : secret "metrics-daemon-secret" not found
Oct 31 13:03:22.221667 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.221447    4660 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 13:03:22.221667 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.221462    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls podName:bdce0f75-852b-49b6-9da3-f662bb82bf94 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:23.221447093 +0000 UTC m=+2.840106240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls") pod "dns-default-pztwr" (UID: "bdce0f75-852b-49b6-9da3-f662bb82bf94") : secret "dns-default-metrics-tls" not found
Oct 31 13:03:22.221667 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.221491    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls podName:aab0721f-f591-4a46-b56e-27848f31642f nodeName:}" failed. No retries permitted until 2026-10-31 13:03:23.22148167 +0000 UTC m=+2.840140819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hjjp" (UID: "aab0721f-f591-4a46-b56e-27848f31642f") : secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:22.221667 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.221448    4660 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:22.221667 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.221508    4660 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694db45b8d-szb8q: secret "image-registry-tls" not found
Oct 31 13:03:22.221667 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.221509    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls podName:7810b5de-3907-4a6e-b11e-0c631c148438 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:23.221501105 +0000 UTC m=+2.840160274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7sfns" (UID: "7810b5de-3907-4a6e-b11e-0c631c148438") : secret "samples-operator-tls" not found
Oct 31 13:03:22.221667 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.221544    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls podName:298ad120-4782-4b13-985f-5af7fb24d97d nodeName:}" failed. No retries permitted until 2026-10-31 13:03:23.221530992 +0000 UTC m=+2.840190140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls") pod "image-registry-694db45b8d-szb8q" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d") : secret "image-registry-tls" not found
Oct 31 13:03:22.221667 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:22.221597    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:22.221667 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:22.221636    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:22.221667 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.221663    4660 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:22.222173 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.221722    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls podName:bf8d1062-0bb5-46f2-98bf-e4d52689294c nodeName:}" failed. No retries permitted until 2026-10-31 13:03:23.221705303 +0000 UTC m=+2.840364449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-44698" (UID: "bf8d1062-0bb5-46f2-98bf-e4d52689294c") : secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:22.222173 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.221768    4660 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 13:03:22.222173 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.221814    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert podName:73811006-7011-4339-800d-5f1b8a9bbf88 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:23.221802542 +0000 UTC m=+2.840461704 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-swrc7" (UID: "73811006-7011-4339-800d-5f1b8a9bbf88") : secret "networking-console-plugin-cert" not found
Oct 31 13:03:22.574622 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:22.574555    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:03:22.575338 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:22.574977    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/1.log"
Oct 31 13:03:22.575338 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:22.575016    4660 generic.go:356] "Generic (PLEG): container finished" podID="4504b20c-a7d8-49e9-a572-9004fce4af1d" containerID="7e4c4db00d89c53443b975a6d3f6ff99bc50be8218581288f9d8bb50a91c1645" exitCode=255
Oct 31 13:03:22.575338 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:22.575083    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9" event={"ID":"4504b20c-a7d8-49e9-a572-9004fce4af1d","Type":"ContainerDied","Data":"7e4c4db00d89c53443b975a6d3f6ff99bc50be8218581288f9d8bb50a91c1645"}
Oct 31 13:03:22.575338 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:22.575141    4660 scope.go:122] "RemoveContainer" containerID="3d8b55e30b935d5d939f5ab6056e8c94bd638608bc9e562793f30465208bd095"
Oct 31 13:03:22.575648 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:22.575632    4660 scope.go:122] "RemoveContainer" containerID="7e4c4db00d89c53443b975a6d3f6ff99bc50be8218581288f9d8bb50a91c1645"
Oct 31 13:03:22.575807 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:22.575789    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7v9w9_openshift-console-operator(4504b20c-a7d8-49e9-a572-9004fce4af1d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9" podUID="4504b20c-a7d8-49e9-a572-9004fce4af1d"
Oct 31 13:03:23.113899 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:23.113842    4660 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mttz6_dc5cb602-ea83-47e3-8966-b832277cb06c/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:23.130929 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:23.130904    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:23.131062 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:23.130953    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:23.131062 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.131029    4660 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:23.131062 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.131038    4660 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76fcd8cbb5-tmzzx: secret "image-registry-tls" not found
Oct 31 13:03:23.131224 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.131068    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.131049239 +0000 UTC m=+4.749708395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : configmap references non-existent config key: service-ca.crt
Oct 31 13:03:23.131224 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.131095    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls podName:7620273c-a4f1-4e1f-ba26-453913b2e171 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.131084711 +0000 UTC m=+4.749743853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls") pod "image-registry-76fcd8cbb5-tmzzx" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171") : secret "image-registry-tls" not found
Oct 31 13:03:23.131348 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:23.131252    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:23.131420 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.131400    4660 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 13:03:23.131472 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.131461    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.131445925 +0000 UTC m=+4.750105088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : secret "router-metrics-certs-default" not found
Oct 31 13:03:23.232736 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:23.232704    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:23.232736 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:23.232739    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:23.232969 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:23.232778    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:23.232969 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.232844    4660 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 13:03:23.232969 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.232855    4660 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:23.232969 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.232861    4660 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:23.232969 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.232878    4660 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694db45b8d-szb8q: secret "image-registry-tls" not found
Oct 31 13:03:23.232969 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.232903    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls podName:7810b5de-3907-4a6e-b11e-0c631c148438 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.23288932 +0000 UTC m=+4.851548461 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7sfns" (UID: "7810b5de-3907-4a6e-b11e-0c631c148438") : secret "samples-operator-tls" not found
Oct 31 13:03:23.232969 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:23.232932    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:23.233368 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.232981    4660 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 13:03:23.233368 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.233027    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert podName:73811006-7011-4339-800d-5f1b8a9bbf88 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.233010714 +0000 UTC m=+4.851669864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-swrc7" (UID: "73811006-7011-4339-800d-5f1b8a9bbf88") : secret "networking-console-plugin-cert" not found
Oct 31 13:03:23.233368 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.233057    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls podName:bf8d1062-0bb5-46f2-98bf-e4d52689294c nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.23304719 +0000 UTC m=+4.851706345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-44698" (UID: "bf8d1062-0bb5-46f2-98bf-e4d52689294c") : secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:23.233368 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.233080    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls podName:298ad120-4782-4b13-985f-5af7fb24d97d nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.233071342 +0000 UTC m=+4.851730490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls") pod "image-registry-694db45b8d-szb8q" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d") : secret "image-registry-tls" not found
Oct 31 13:03:23.233368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:23.233115    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:23.233368 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.233207    4660 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 13:03:23.233368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:23.233166    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:23.233368 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.233239    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs podName:cf350772-aff8-4051-820a-dd114e01e441 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.23323031 +0000 UTC m=+4.851889476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs") pod "network-metrics-daemon-pphvg" (UID: "cf350772-aff8-4051-820a-dd114e01e441") : secret "metrics-daemon-secret" not found
Oct 31 13:03:23.233368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:23.233271    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:23.233368 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.233314    4660 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:23.233368 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.233349    4660 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 13:03:23.233368 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.233357    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls podName:aab0721f-f591-4a46-b56e-27848f31642f nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.233342962 +0000 UTC m=+4.852002118 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hjjp" (UID: "aab0721f-f591-4a46-b56e-27848f31642f") : secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:23.233739 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:23.233384    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls podName:bdce0f75-852b-49b6-9da3-f662bb82bf94 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.233374845 +0000 UTC m=+4.852033987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls") pod "dns-default-pztwr" (UID: "bdce0f75-852b-49b6-9da3-f662bb82bf94") : secret "dns-default-metrics-tls" not found
Oct 31 13:03:23.280006 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:23.279982    4660 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mttz6_dc5cb602-ea83-47e3-8966-b832277cb06c/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:23.453619 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:23.453586    4660 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mttz6_dc5cb602-ea83-47e3-8966-b832277cb06c/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:23.579669 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:23.579645    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:03:23.606061 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:23.606039    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:23.635207 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:23.635178    4660 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mttz6_dc5cb602-ea83-47e3-8966-b832277cb06c/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:24.215214 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:24.215185    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gpl2l_7231ca35-6551-41ea-b341-46f8836425c1/dns-node-resolver/0.log"
Oct 31 13:03:24.272040 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:24.272016    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:24.582985 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:24.582916    4660 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 13:03:24.735931 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:24.735903    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:03:25.152498 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.152464    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:25.152668 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.152566    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:25.152668 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.152596    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:25.152668 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:25.152611    4660 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:25.152668 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:25.152633    4660 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76fcd8cbb5-tmzzx: secret "image-registry-tls" not found
Oct 31 13:03:25.152792 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:25.152699    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls podName:7620273c-a4f1-4e1f-ba26-453913b2e171 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:29.152680803 +0000 UTC m=+8.771339958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls") pod "image-registry-76fcd8cbb5-tmzzx" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171") : secret "image-registry-tls" not found
Oct 31 13:03:25.152792 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:25.152698    4660 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 13:03:25.152792 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:25.152740    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:29.15272915 +0000 UTC m=+8.771388305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : secret "router-metrics-certs-default" not found
Oct 31 13:03:25.153168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.153151    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95a45a3-7757-4843-9161-6cdbf64385cd-service-ca-bundle\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:25.253261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.253226    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:25.253443 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.253269    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:25.253443 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.253347    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:25.253443 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:25.253369    4660 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:25.253443 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.253384    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:25.253443 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.253425    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:25.253700 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:25.253445    4660 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 13:03:25.253700 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.253468    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:25.253700 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:25.253498    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert podName:73811006-7011-4339-800d-5f1b8a9bbf88 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:29.253481817 +0000 UTC m=+8.872140962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-swrc7" (UID: "73811006-7011-4339-800d-5f1b8a9bbf88") : secret "networking-console-plugin-cert" not found
Oct 31 13:03:25.253700 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:25.253554    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls podName:bf8d1062-0bb5-46f2-98bf-e4d52689294c nodeName:}" failed. No retries permitted until 2026-10-31 13:03:29.253523508 +0000 UTC m=+8.872182650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-44698" (UID: "bf8d1062-0bb5-46f2-98bf-e4d52689294c") : secret "cluster-monitoring-operator-tls" not found
Oct 31 13:03:25.253700 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:25.253560    4660 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 13:03:25.253700 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.253574    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:25.253700 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:25.253606    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls podName:7810b5de-3907-4a6e-b11e-0c631c148438 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:29.253591255 +0000 UTC m=+8.872250416 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7sfns" (UID: "7810b5de-3907-4a6e-b11e-0c631c148438") : secret "samples-operator-tls" not found
Oct 31 13:03:25.253700 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:25.253632    4660 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 13:03:25.253700 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:25.253641    4660 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-694db45b8d-szb8q: secret "image-registry-tls" not found
Oct 31 13:03:25.253700 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:25.253672    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls podName:298ad120-4782-4b13-985f-5af7fb24d97d nodeName:}" failed. No retries permitted until 2026-10-31 13:03:29.253663709 +0000 UTC m=+8.872322851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls") pod "image-registry-694db45b8d-szb8q" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d") : secret "image-registry-tls" not found
Oct 31 13:03:25.254057 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.254037    4660 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 13:03:25.257486 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.257459    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab0721f-f591-4a46-b56e-27848f31642f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hjjp\" (UID: \"aab0721f-f591-4a46-b56e-27848f31642f\") " pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:25.257589 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.257501    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf350772-aff8-4051-820a-dd114e01e441-metrics-certs\") pod \"network-metrics-daemon-pphvg\" (UID: \"cf350772-aff8-4051-820a-dd114e01e441\") " pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:25.257589 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.257516    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdce0f75-852b-49b6-9da3-f662bb82bf94-metrics-tls\") pod \"dns-default-pztwr\" (UID: \"bdce0f75-852b-49b6-9da3-f662bb82bf94\") " pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:25.397654 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.397627    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pphvg"
Oct 31 13:03:25.399805 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.399786    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:25.400863 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.400848    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9hjjp"
Oct 31 13:03:25.416041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.416014    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lgcv8_41b93592-66ac-4e0e-acf9-11271af65e89/node-ca/0.log"
Oct 31 13:03:25.701817 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.701750    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:25.702153 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.702084    4660 scope.go:122] "RemoveContainer" containerID="7e4c4db00d89c53443b975a6d3f6ff99bc50be8218581288f9d8bb50a91c1645"
Oct 31 13:03:25.702272 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:25.702254    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7v9w9_openshift-console-operator(4504b20c-a7d8-49e9-a572-9004fce4af1d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9" podUID="4504b20c-a7d8-49e9-a572-9004fce4af1d"
Oct 31 13:03:25.708400 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.708226    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pphvg"]
Oct 31 13:03:25.708400 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.708273    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pztwr"]
Oct 31 13:03:25.708695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.708647    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9hjjp"]
Oct 31 13:03:25.710963 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:25.710941    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf350772_aff8_4051_820a_dd114e01e441.slice/crio-05ff9eaba1844df350b1d0830beda85e1db86cbefdc2ef334fa2ed406025df24 WatchSource:0}: Error finding container 05ff9eaba1844df350b1d0830beda85e1db86cbefdc2ef334fa2ed406025df24: Status 404 returned error can't find the container with id 05ff9eaba1844df350b1d0830beda85e1db86cbefdc2ef334fa2ed406025df24
Oct 31 13:03:25.711672 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:25.711653    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdce0f75_852b_49b6_9da3_f662bb82bf94.slice/crio-99439f01d9c7ce852be637678a6c56cf3ff0cbb595060006d597b7cf5db88be0 WatchSource:0}: Error finding container 99439f01d9c7ce852be637678a6c56cf3ff0cbb595060006d597b7cf5db88be0: Status 404 returned error can't find the container with id 99439f01d9c7ce852be637678a6c56cf3ff0cbb595060006d597b7cf5db88be0
Oct 31 13:03:25.712426 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:25.712409    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab0721f_f591_4a46_b56e_27848f31642f.slice/crio-bd484d59b19467d9fc65c7ef433be7560dfa2b4795d250b7e13f223511993fd2 WatchSource:0}: Error finding container bd484d59b19467d9fc65c7ef433be7560dfa2b4795d250b7e13f223511993fd2: Status 404 returned error can't find the container with id bd484d59b19467d9fc65c7ef433be7560dfa2b4795d250b7e13f223511993fd2
Oct 31 13:03:25.713909 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:25.713890    4660 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:03:26.416733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:26.416698    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l42ds_bd718975-d0cb-4a21-8383-c29164ae4f31/serve-healthcheck-canary/0.log"
Oct 31 13:03:26.591922 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:26.591897    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pphvg" event={"ID":"cf350772-aff8-4051-820a-dd114e01e441","Type":"ContainerStarted","Data":"05ff9eaba1844df350b1d0830beda85e1db86cbefdc2ef334fa2ed406025df24"}
Oct 31 13:03:26.593971 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:26.593946    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9hjjp" event={"ID":"aab0721f-f591-4a46-b56e-27848f31642f","Type":"ContainerStarted","Data":"a184b3ac541dc8c317d29560032af6a994d483e809b3f977f097fc8375ff0270"}
Oct 31 13:03:26.594124 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:26.594008    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9hjjp" event={"ID":"aab0721f-f591-4a46-b56e-27848f31642f","Type":"ContainerStarted","Data":"bd484d59b19467d9fc65c7ef433be7560dfa2b4795d250b7e13f223511993fd2"}
Oct 31 13:03:26.595182 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:26.595161    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pztwr" event={"ID":"bdce0f75-852b-49b6-9da3-f662bb82bf94","Type":"ContainerStarted","Data":"99439f01d9c7ce852be637678a6c56cf3ff0cbb595060006d597b7cf5db88be0"}
Oct 31 13:03:27.018072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:27.018043    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s_f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a/kube-storage-version-migrator-operator/0.log"
Oct 31 13:03:27.601701 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:27.601654    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pphvg" event={"ID":"cf350772-aff8-4051-820a-dd114e01e441","Type":"ContainerStarted","Data":"024e6a096bfabe6eb20f6e516a8c1fcd450c94f1023da2cd68be6e1cbcc055a4"}
Oct 31 13:03:27.601902 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:27.601725    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pphvg" event={"ID":"cf350772-aff8-4051-820a-dd114e01e441","Type":"ContainerStarted","Data":"3af38093d545447b4c7e3d6208287151911a00f09ebfc1655ef0b83256d40427"}
Oct 31 13:03:27.603933 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:27.603907    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9hjjp" event={"ID":"aab0721f-f591-4a46-b56e-27848f31642f","Type":"ContainerStarted","Data":"f7df20392ec75cecc41dbfb85e47e4ffc228366be417046d2b8c7d6ff3a42a3f"}
Oct 31 13:03:27.870308 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:27.870262    4660 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:27.870783 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:27.870766    4660 scope.go:122] "RemoveContainer" containerID="7e4c4db00d89c53443b975a6d3f6ff99bc50be8218581288f9d8bb50a91c1645"
Oct 31 13:03:27.871014 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:27.870991    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7v9w9_openshift-console-operator(4504b20c-a7d8-49e9-a572-9004fce4af1d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9" podUID="4504b20c-a7d8-49e9-a572-9004fce4af1d"
Oct 31 13:03:28.608884 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:28.608843    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pztwr" event={"ID":"bdce0f75-852b-49b6-9da3-f662bb82bf94","Type":"ContainerStarted","Data":"290bb51d7e83723cfb0844b896480774484f83e81438fbfb4e2a2e7be5930973"}
Oct 31 13:03:28.609362 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:28.608950    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pztwr" event={"ID":"bdce0f75-852b-49b6-9da3-f662bb82bf94","Type":"ContainerStarted","Data":"6e2a72ccf3e830a79a5953f56d7ab4879314c42553494ea41539b0af11ec9d46"}
Oct 31 13:03:28.609362 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:28.608982    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:28.610979 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:28.610955    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9hjjp" event={"ID":"aab0721f-f591-4a46-b56e-27848f31642f","Type":"ContainerStarted","Data":"91b4a6c9aa7d03e261b7b9e561e4f13aaf09f56d00eda0cd56b2084047921a13"}
Oct 31 13:03:29.189332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.189287    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:29.189522 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.189393    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:29.189522 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:29.189480    4660 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 13:03:29.189615 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:29.189542    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs podName:c95a45a3-7757-4843-9161-6cdbf64385cd nodeName:}" failed. No retries permitted until 2026-10-31 13:03:37.189528465 +0000 UTC m=+16.808187621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs") pod "router-default-848dcf9575-7trg4" (UID: "c95a45a3-7757-4843-9161-6cdbf64385cd") : secret "router-metrics-certs-default" not found
Oct 31 13:03:29.191669 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.191626    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls\") pod \"image-registry-76fcd8cbb5-tmzzx\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") " pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:29.284951 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.284921    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:29.290267 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.290247    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:29.290351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.290281    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:29.290428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.290387    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:29.290428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.290414    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:29.290527 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:29.290425    4660 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 13:03:29.290527 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:29.290514    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert podName:73811006-7011-4339-800d-5f1b8a9bbf88 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:37.290493016 +0000 UTC m=+16.909152179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-swrc7" (UID: "73811006-7011-4339-800d-5f1b8a9bbf88") : secret "networking-console-plugin-cert" not found
Oct 31 13:03:29.292822 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.292798    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls\") pod \"image-registry-694db45b8d-szb8q\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") " pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:29.292902 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.292807    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7810b5de-3907-4a6e-b11e-0c631c148438-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7sfns\" (UID: \"7810b5de-3907-4a6e-b11e-0c631c148438\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:29.292902 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.292852    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bf8d1062-0bb5-46f2-98bf-e4d52689294c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-44698\" (UID: \"bf8d1062-0bb5-46f2-98bf-e4d52689294c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:29.299685 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.299652    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:29.565533 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.565460    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"]
Oct 31 13:03:29.568891 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:29.568870    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7620273c_a4f1_4e1f_ba26_453913b2e171.slice/crio-b1cddf6ad3b73ea71c7575a736ce8bbe93e0696d0dac5624907099a711a3d8fa WatchSource:0}: Error finding container b1cddf6ad3b73ea71c7575a736ce8bbe93e0696d0dac5624907099a711a3d8fa: Status 404 returned error can't find the container with id b1cddf6ad3b73ea71c7575a736ce8bbe93e0696d0dac5624907099a711a3d8fa
Oct 31 13:03:29.584060 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.584035    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-694db45b8d-szb8q"]
Oct 31 13:03:29.584197 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.584177    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698"
Oct 31 13:03:29.584536 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.584519    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"
Oct 31 13:03:29.616390 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.616359    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-694db45b8d-szb8q" event={"ID":"298ad120-4782-4b13-985f-5af7fb24d97d","Type":"ContainerStarted","Data":"d3acd803c1d2a015c837961abcc7698d77031bd0640113848f2e13768e7fbb63"}
Oct 31 13:03:29.618372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.618341    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx" event={"ID":"7620273c-a4f1-4e1f-ba26-453913b2e171","Type":"ContainerStarted","Data":"b1cddf6ad3b73ea71c7575a736ce8bbe93e0696d0dac5624907099a711a3d8fa"}
Oct 31 13:03:29.877985 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.877956    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-44698"]
Oct 31 13:03:29.881026 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:29.881002    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns"]
Oct 31 13:03:29.881990 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:29.881913    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8d1062_0bb5_46f2_98bf_e4d52689294c.slice/crio-8ec776805b12d3bf6108a813ce0208e225a1b21ef298f88121849ee48d521cc2 WatchSource:0}: Error finding container 8ec776805b12d3bf6108a813ce0208e225a1b21ef298f88121849ee48d521cc2: Status 404 returned error can't find the container with id 8ec776805b12d3bf6108a813ce0208e225a1b21ef298f88121849ee48d521cc2
Oct 31 13:03:30.623622 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:30.623578    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-694db45b8d-szb8q" event={"ID":"298ad120-4782-4b13-985f-5af7fb24d97d","Type":"ContainerStarted","Data":"4a651f0a59ca5442005a292376e368e3b7306e113cf483c2331454a75520bcc1"}
Oct 31 13:03:30.625161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:30.625120    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns" event={"ID":"7810b5de-3907-4a6e-b11e-0c631c148438","Type":"ContainerStarted","Data":"37e12af134a051018e7b0e317203389a9d520a45bcebec19bdcec1f2dcdc3f7d"}
Oct 31 13:03:30.626472 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:30.626444    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698" event={"ID":"bf8d1062-0bb5-46f2-98bf-e4d52689294c","Type":"ContainerStarted","Data":"8ec776805b12d3bf6108a813ce0208e225a1b21ef298f88121849ee48d521cc2"}
Oct 31 13:03:30.628074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:30.628047    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx" event={"ID":"7620273c-a4f1-4e1f-ba26-453913b2e171","Type":"ContainerStarted","Data":"e210f4f0ae99573690c9e7a36dba1184e63b2a616ed2008eefa7d213e5c133f1"}
Oct 31 13:03:31.391104 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:31.391072    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7nv8r"
Oct 31 13:03:31.632537 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:31.632503    4660 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 13:03:31.633135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:31.633115    4660 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 13:03:32.638561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:32.638514    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns" event={"ID":"7810b5de-3907-4a6e-b11e-0c631c148438","Type":"ContainerStarted","Data":"a96f80459eca2739fbea6152ccfac05cfb143843bb3b15a87c55bb226db29b9c"}
Oct 31 13:03:32.638997 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:32.638583    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7sfns" event={"ID":"7810b5de-3907-4a6e-b11e-0c631c148438","Type":"ContainerStarted","Data":"2ee3b4095e19fc62b8edfa8df95c7b27f40ac233be93928c69830f24f64ee633"}
Oct 31 13:03:32.640034 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:32.639996    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-44698" event={"ID":"bf8d1062-0bb5-46f2-98bf-e4d52689294c","Type":"ContainerStarted","Data":"5a9f405fc014ee4afee12eb01fdcbabb55b546c0282d347c44c78c2781d0c873"}
Oct 31 13:03:33.520039 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:33.520010    4660 scope.go:122] "RemoveContainer" containerID="7e4c4db00d89c53443b975a6d3f6ff99bc50be8218581288f9d8bb50a91c1645"
Oct 31 13:03:33.645696 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:33.645668    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:03:33.646042 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:33.645780    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9" event={"ID":"4504b20c-a7d8-49e9-a572-9004fce4af1d","Type":"ContainerStarted","Data":"1fe7a71859fe54d190900a0c43fd922779724448ffbd2c81f5aafca395f87fa7"}
Oct 31 13:03:33.646327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:33.646071    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:33.800818 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:33.800752    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7v9w9"
Oct 31 13:03:36.272771 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:36.272737    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:36.402778 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:36.402739    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:03:37.262699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:37.262660    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:37.265022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:37.264997    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c95a45a3-7757-4843-9161-6cdbf64385cd-metrics-certs\") pod \"router-default-848dcf9575-7trg4\" (UID: \"c95a45a3-7757-4843-9161-6cdbf64385cd\") " pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:37.363908 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:37.363878    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:37.366147 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:37.366128    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73811006-7011-4339-800d-5f1b8a9bbf88-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-swrc7\" (UID: \"73811006-7011-4339-800d-5f1b8a9bbf88\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:37.383663 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:37.383634    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:37.387827 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:37.387805    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7"
Oct 31 13:03:37.661736 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:37.661665    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-swrc7"]
Oct 31 13:03:37.667395 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:37.667366    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73811006_7011_4339_800d_5f1b8a9bbf88.slice/crio-47b2831b0d52abe3f591d8bfa91837f3de56e27547bfdc2ea37bc29acd703855 WatchSource:0}: Error finding container 47b2831b0d52abe3f591d8bfa91837f3de56e27547bfdc2ea37bc29acd703855: Status 404 returned error can't find the container with id 47b2831b0d52abe3f591d8bfa91837f3de56e27547bfdc2ea37bc29acd703855
Oct 31 13:03:37.668358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:37.668194    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-848dcf9575-7trg4"]
Oct 31 13:03:37.670984 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:37.670965    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95a45a3_7757_4843_9161_6cdbf64385cd.slice/crio-93c1b44a36bdb487fa88a14955e2e17a4a39a61aad9e9eb928a90248ffa3dbc9 WatchSource:0}: Error finding container 93c1b44a36bdb487fa88a14955e2e17a4a39a61aad9e9eb928a90248ffa3dbc9: Status 404 returned error can't find the container with id 93c1b44a36bdb487fa88a14955e2e17a4a39a61aad9e9eb928a90248ffa3dbc9
Oct 31 13:03:38.621628 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:38.621592    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pztwr"
Oct 31 13:03:38.664850 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:38.664811    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7" event={"ID":"73811006-7011-4339-800d-5f1b8a9bbf88","Type":"ContainerStarted","Data":"47b2831b0d52abe3f591d8bfa91837f3de56e27547bfdc2ea37bc29acd703855"}
Oct 31 13:03:38.666204 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:38.666182    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-848dcf9575-7trg4" event={"ID":"c95a45a3-7757-4843-9161-6cdbf64385cd","Type":"ContainerStarted","Data":"efdc228634bd600e64917c8e4fdbd3dbe9976349555e0cfbdb79c210c7974f00"}
Oct 31 13:03:38.666273 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:38.666219    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-848dcf9575-7trg4" event={"ID":"c95a45a3-7757-4843-9161-6cdbf64385cd","Type":"ContainerStarted","Data":"93c1b44a36bdb487fa88a14955e2e17a4a39a61aad9e9eb928a90248ffa3dbc9"}
Oct 31 13:03:39.384363 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:39.384332    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:39.386589 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:39.386567    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:39.672143 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:39.672062    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-swrc7" event={"ID":"73811006-7011-4339-800d-5f1b8a9bbf88","Type":"ContainerStarted","Data":"1b15a5c88fd2402ddd39ccf7e4c58c294b553baab864423f97107a9f9862a740"}
Oct 31 13:03:39.672555 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:39.672320    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:39.673399 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:39.673378    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-848dcf9575-7trg4"
Oct 31 13:03:40.396164 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:40.396127    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"]
Oct 31 13:03:45.769453 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:45.769418    4660 patch_prober.go:29] interesting pod/image-registry-694db45b8d-szb8q container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 31 13:03:45.769937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:45.769478    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-694db45b8d-szb8q" podUID="298ad120-4782-4b13-985f-5af7fb24d97d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:03:46.407358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:46.407321    4660 patch_prober.go:29] interesting pod/image-registry-694db45b8d-szb8q container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 31 13:03:46.407544 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:46.407373    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-694db45b8d-szb8q" podUID="298ad120-4782-4b13-985f-5af7fb24d97d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:03:47.530924 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.530887    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rgjfb"]
Oct 31 13:03:47.536994 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.536970    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.577610 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.577582    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 13:03:47.577610 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.577583    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 13:03:47.577796 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.577704    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rrldl\""
Oct 31 13:03:47.595195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.595169    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 13:03:47.595195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.595169    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 13:03:47.648742 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.648713    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdz2w\" (UniqueName: \"kubernetes.io/projected/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-kube-api-access-hdz2w\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.648934 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.648751    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-node-exporter-wtmp\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.648934 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.648772    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-root\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.648934 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.648856    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-node-exporter-textfile\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.648934 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.648893    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-node-exporter-accelerators-collector-config\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.649108 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.648955    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-node-exporter-tls\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.649108 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.648982    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.649108 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.649003    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-sys\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.649108 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.649062    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-metrics-client-ca\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.750369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.750332    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-node-exporter-tls\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.750369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.750372    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.750613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.750391    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-sys\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.750613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.750417    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-metrics-client-ca\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.750613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.750457    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdz2w\" (UniqueName: \"kubernetes.io/projected/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-kube-api-access-hdz2w\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.750613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.750480    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-node-exporter-wtmp\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.750613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.750509    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-root\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.750613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.750527    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-sys\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.750613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.750538    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-node-exporter-textfile\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.750613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.750581    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-root\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.751088 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.750652    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-node-exporter-accelerators-collector-config\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.751088 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.750861    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-node-exporter-wtmp\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.751088 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.750889    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-node-exporter-textfile\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.751239 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.751192    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-node-exporter-accelerators-collector-config\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.751239 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.751194    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-metrics-client-ca\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.752849 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.752828    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.752952 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.752899    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-node-exporter-tls\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.763166 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.763144    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdz2w\" (UniqueName: \"kubernetes.io/projected/4e43df86-2b4a-4deb-ac8f-d39ba65c9f73-kube-api-access-hdz2w\") pod \"node-exporter-rgjfb\" (UID: \"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73\") " pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.857390 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:47.857321    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rgjfb"
Oct 31 13:03:47.871029 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:03:47.871001    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e43df86_2b4a_4deb_ac8f_d39ba65c9f73.slice/crio-18688098c87e584cbf771af42bb84169ef084949ba9b34a8e08d90e03986ec97 WatchSource:0}: Error finding container 18688098c87e584cbf771af42bb84169ef084949ba9b34a8e08d90e03986ec97: Status 404 returned error can't find the container with id 18688098c87e584cbf771af42bb84169ef084949ba9b34a8e08d90e03986ec97
Oct 31 13:03:48.704708 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:48.704670    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgjfb" event={"ID":"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73","Type":"ContainerStarted","Data":"18688098c87e584cbf771af42bb84169ef084949ba9b34a8e08d90e03986ec97"}
Oct 31 13:03:49.710772 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:49.710737    4660 generic.go:356] "Generic (PLEG): container finished" podID="4e43df86-2b4a-4deb-ac8f-d39ba65c9f73" containerID="11ddffa9fbce4fc928d26790baa0838570671b4e64e28e8fedd9423ed50716ce" exitCode=0
Oct 31 13:03:49.711109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:49.710804    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgjfb" event={"ID":"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73","Type":"ContainerDied","Data":"11ddffa9fbce4fc928d26790baa0838570671b4e64e28e8fedd9423ed50716ce"}
Oct 31 13:03:50.402626 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:50.402594    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:03:50.716162 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:50.716084    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgjfb" event={"ID":"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73","Type":"ContainerStarted","Data":"e27c14f02593a8214335ec292d59725b0471de3ccff633c2eb58e5b6775f7fdc"}
Oct 31 13:03:50.716162 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:50.716148    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgjfb" event={"ID":"4e43df86-2b4a-4deb-ac8f-d39ba65c9f73","Type":"ContainerStarted","Data":"10d47eecb6e635d7638f587a0ecec68ba96223cdee842cc92996b4c2e22878f6"}
Oct 31 13:03:50.730706 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:03:50.730674    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:03:50.740321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:50.740142    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rgjfb" podStartSLOduration=2.049406226 podStartE2EDuration="3.740125533s" totalImagesPullingTime="1.61408421s" totalInitContainerRuntime="76.635097ms" isStatefulPod=true podCreationTimestamp="2026-10-31 13:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:50.739324242 +0000 UTC m=+30.357983406" watchObservedRunningTime="2026-10-31 13:03:50.740125533 +0000 UTC m=+30.358784699"
Oct 31 13:03:51.731479 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:51.731452    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mttz6"
Oct 31 13:03:54.738411 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:54.738376    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tkc46"
Oct 31 13:03:56.407132 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:03:56.407103    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:04:02.932728 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:02.932611    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-694db45b8d-szb8q"]
Oct 31 13:04:05.417801 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.417748    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx" podUID="7620273c-a4f1-4e1f-ba26-453913b2e171" containerName="registry" containerID="cri-o://e210f4f0ae99573690c9e7a36dba1184e63b2a616ed2008eefa7d213e5c133f1" gracePeriod=30
Oct 31 13:04:05.674547 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.674486    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:04:05.774272 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.774232    4660 generic.go:356] "Generic (PLEG): container finished" podID="7620273c-a4f1-4e1f-ba26-453913b2e171" containerID="e210f4f0ae99573690c9e7a36dba1184e63b2a616ed2008eefa7d213e5c133f1" exitCode=0
Oct 31 13:04:05.774474 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.774278    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"
Oct 31 13:04:05.774474 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.774322    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx" event={"ID":"7620273c-a4f1-4e1f-ba26-453913b2e171","Type":"ContainerDied","Data":"e210f4f0ae99573690c9e7a36dba1184e63b2a616ed2008eefa7d213e5c133f1"}
Oct 31 13:04:05.774474 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.774401    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76fcd8cbb5-tmzzx" event={"ID":"7620273c-a4f1-4e1f-ba26-453913b2e171","Type":"ContainerDied","Data":"b1cddf6ad3b73ea71c7575a736ce8bbe93e0696d0dac5624907099a711a3d8fa"}
Oct 31 13:04:05.774474 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.774425    4660 scope.go:122] "RemoveContainer" containerID="e210f4f0ae99573690c9e7a36dba1184e63b2a616ed2008eefa7d213e5c133f1"
Oct 31 13:04:05.782443 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.782399    4660 scope.go:122] "RemoveContainer" containerID="e210f4f0ae99573690c9e7a36dba1184e63b2a616ed2008eefa7d213e5c133f1"
Oct 31 13:04:05.782680 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:04:05.782661    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e210f4f0ae99573690c9e7a36dba1184e63b2a616ed2008eefa7d213e5c133f1\": container with ID starting with e210f4f0ae99573690c9e7a36dba1184e63b2a616ed2008eefa7d213e5c133f1 not found: ID does not exist" containerID="e210f4f0ae99573690c9e7a36dba1184e63b2a616ed2008eefa7d213e5c133f1"
Oct 31 13:04:05.782724 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.782691    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e210f4f0ae99573690c9e7a36dba1184e63b2a616ed2008eefa7d213e5c133f1"} err="failed to get container status \"e210f4f0ae99573690c9e7a36dba1184e63b2a616ed2008eefa7d213e5c133f1\": rpc error: code = NotFound desc = could not find container \"e210f4f0ae99573690c9e7a36dba1184e63b2a616ed2008eefa7d213e5c133f1\": container with ID starting with e210f4f0ae99573690c9e7a36dba1184e63b2a616ed2008eefa7d213e5c133f1 not found: ID does not exist"
Oct 31 13:04:05.811416 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.811398    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-certificates\") pod \"7620273c-a4f1-4e1f-ba26-453913b2e171\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") "
Oct 31 13:04:05.811576 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.811433    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-bound-sa-token\") pod \"7620273c-a4f1-4e1f-ba26-453913b2e171\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") "
Oct 31 13:04:05.811576 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.811457    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-installation-pull-secrets\") pod \"7620273c-a4f1-4e1f-ba26-453913b2e171\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") "
Oct 31 13:04:05.811576 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.811493    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-trusted-ca\") pod \"7620273c-a4f1-4e1f-ba26-453913b2e171\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") "
Oct 31 13:04:05.811576 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.811519    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls\") pod \"7620273c-a4f1-4e1f-ba26-453913b2e171\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") "
Oct 31 13:04:05.811576 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.811550    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7620273c-a4f1-4e1f-ba26-453913b2e171-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7620273c-a4f1-4e1f-ba26-453913b2e171-ca-trust-extracted\") pod \"7620273c-a4f1-4e1f-ba26-453913b2e171\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") "
Oct 31 13:04:05.811824 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.811618    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-image-registry-private-configuration\") pod \"7620273c-a4f1-4e1f-ba26-453913b2e171\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") "
Oct 31 13:04:05.811824 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.811650    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-kube-api-access-mnzhh\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-kube-api-access-mnzhh\") pod \"7620273c-a4f1-4e1f-ba26-453913b2e171\" (UID: \"7620273c-a4f1-4e1f-ba26-453913b2e171\") "
Oct 31 13:04:05.811931 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.811903    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-certificates" pod "7620273c-a4f1-4e1f-ba26-453913b2e171" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:04:05.812101 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.812059    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-trusted-ca" pod "7620273c-a4f1-4e1f-ba26-453913b2e171" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:04:05.813878 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.813850    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-bound-sa-token" pod "7620273c-a4f1-4e1f-ba26-453913b2e171" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:04:05.814217 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.814182    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-installation-pull-secrets" pod "7620273c-a4f1-4e1f-ba26-453913b2e171" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:04:05.814217 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.814183    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls" pod "7620273c-a4f1-4e1f-ba26-453913b2e171" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:04:05.814518 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.814496    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-kube-api-access-mnzhh" pod "7620273c-a4f1-4e1f-ba26-453913b2e171" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171"). InnerVolumeSpecName "kube-api-access-mnzhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:04:05.814518 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.814495    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-image-registry-private-configuration" pod "7620273c-a4f1-4e1f-ba26-453913b2e171" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:04:05.818165 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.818141    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7620273c-a4f1-4e1f-ba26-453913b2e171-ca-trust-extracted" pod "7620273c-a4f1-4e1f-ba26-453913b2e171" (UID: "7620273c-a4f1-4e1f-ba26-453913b2e171"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:04:05.912485 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.912460    4660 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-certificates\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:05.912485 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.912482    4660 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-bound-sa-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:05.912619 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.912492    4660 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-installation-pull-secrets\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:05.912619 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.912502    4660 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7620273c-a4f1-4e1f-ba26-453913b2e171-trusted-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:05.912619 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.912511    4660 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-registry-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:05.912619 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.912518    4660 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7620273c-a4f1-4e1f-ba26-453913b2e171-ca-trust-extracted\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:05.912619 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.912528    4660 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7620273c-a4f1-4e1f-ba26-453913b2e171-image-registry-private-configuration\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:05.912619 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:05.912537    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnzhh\" (UniqueName: \"kubernetes.io/projected/7620273c-a4f1-4e1f-ba26-453913b2e171-kube-api-access-mnzhh\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:06.097178 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:06.097142    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"]
Oct 31 13:04:06.103401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:06.103380    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76fcd8cbb5-tmzzx"]
Oct 31 13:04:06.524173 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:06.524131    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7620273c-a4f1-4e1f-ba26-453913b2e171" path="/var/lib/kubelet/pods/7620273c-a4f1-4e1f-ba26-453913b2e171/volumes"
Oct 31 13:04:20.730010 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:04:20.729973    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:04:27.953705 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:27.953657    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-694db45b8d-szb8q" podUID="298ad120-4782-4b13-985f-5af7fb24d97d" containerName="registry" containerID="cri-o://4a651f0a59ca5442005a292376e368e3b7306e113cf483c2331454a75520bcc1" gracePeriod=30
Oct 31 13:04:28.190678 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.190649    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:04:28.280063 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.279992    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298ad120-4782-4b13-985f-5af7fb24d97d-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/298ad120-4782-4b13-985f-5af7fb24d97d-ca-trust-extracted\") pod \"298ad120-4782-4b13-985f-5af7fb24d97d\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") "
Oct 31 13:04:28.280063 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.280027    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-kube-api-access-gf2v9\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-kube-api-access-gf2v9\") pod \"298ad120-4782-4b13-985f-5af7fb24d97d\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") "
Oct 31 13:04:28.280063 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.280062    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-registry-certificates\") pod \"298ad120-4782-4b13-985f-5af7fb24d97d\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") "
Oct 31 13:04:28.280354 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.280100    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-installation-pull-secrets\") pod \"298ad120-4782-4b13-985f-5af7fb24d97d\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") "
Oct 31 13:04:28.280354 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.280133    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-trusted-ca\") pod \"298ad120-4782-4b13-985f-5af7fb24d97d\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") "
Oct 31 13:04:28.280354 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.280219    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-bound-sa-token\") pod \"298ad120-4782-4b13-985f-5af7fb24d97d\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") "
Oct 31 13:04:28.280354 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.280264    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-image-registry-private-configuration\") pod \"298ad120-4782-4b13-985f-5af7fb24d97d\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") "
Oct 31 13:04:28.280354 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.280318    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls\") pod \"298ad120-4782-4b13-985f-5af7fb24d97d\" (UID: \"298ad120-4782-4b13-985f-5af7fb24d97d\") "
Oct 31 13:04:28.280617 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.280565    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-registry-certificates" pod "298ad120-4782-4b13-985f-5af7fb24d97d" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:04:28.280674 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.280644    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-trusted-ca" pod "298ad120-4782-4b13-985f-5af7fb24d97d" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:04:28.282812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.282779    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-image-registry-private-configuration" pod "298ad120-4782-4b13-985f-5af7fb24d97d" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:04:28.282938 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.282867    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-kube-api-access-gf2v9" pod "298ad120-4782-4b13-985f-5af7fb24d97d" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d"). InnerVolumeSpecName "kube-api-access-gf2v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:04:28.283007 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.282993    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls" pod "298ad120-4782-4b13-985f-5af7fb24d97d" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:04:28.283069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.282993    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-bound-sa-token" pod "298ad120-4782-4b13-985f-5af7fb24d97d" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:04:28.283069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.283013    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-installation-pull-secrets" pod "298ad120-4782-4b13-985f-5af7fb24d97d" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:04:28.286782 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.286757    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298ad120-4782-4b13-985f-5af7fb24d97d-ca-trust-extracted" pod "298ad120-4782-4b13-985f-5af7fb24d97d" (UID: "298ad120-4782-4b13-985f-5af7fb24d97d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:04:28.381528 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.381485    4660 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-trusted-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:28.381528 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.381521    4660 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-bound-sa-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:28.381528 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.381532    4660 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-image-registry-private-configuration\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:28.381528 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.381544    4660 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-registry-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:28.381776 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.381554    4660 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/298ad120-4782-4b13-985f-5af7fb24d97d-ca-trust-extracted\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:28.381776 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.381562    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf2v9\" (UniqueName: \"kubernetes.io/projected/298ad120-4782-4b13-985f-5af7fb24d97d-kube-api-access-gf2v9\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:28.381776 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.381572    4660 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/298ad120-4782-4b13-985f-5af7fb24d97d-registry-certificates\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:28.381776 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.381581    4660 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/298ad120-4782-4b13-985f-5af7fb24d97d-installation-pull-secrets\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:28.861173 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.861094    4660 generic.go:356] "Generic (PLEG): container finished" podID="298ad120-4782-4b13-985f-5af7fb24d97d" containerID="4a651f0a59ca5442005a292376e368e3b7306e113cf483c2331454a75520bcc1" exitCode=0
Oct 31 13:04:28.861173 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.861139    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-694db45b8d-szb8q"
Oct 31 13:04:28.861374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.861141    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-694db45b8d-szb8q" event={"ID":"298ad120-4782-4b13-985f-5af7fb24d97d","Type":"ContainerDied","Data":"4a651f0a59ca5442005a292376e368e3b7306e113cf483c2331454a75520bcc1"}
Oct 31 13:04:28.861374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.861266    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-694db45b8d-szb8q" event={"ID":"298ad120-4782-4b13-985f-5af7fb24d97d","Type":"ContainerDied","Data":"d3acd803c1d2a015c837961abcc7698d77031bd0640113848f2e13768e7fbb63"}
Oct 31 13:04:28.861374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.861307    4660 scope.go:122] "RemoveContainer" containerID="4a651f0a59ca5442005a292376e368e3b7306e113cf483c2331454a75520bcc1"
Oct 31 13:04:28.868990 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.868974    4660 scope.go:122] "RemoveContainer" containerID="4a651f0a59ca5442005a292376e368e3b7306e113cf483c2331454a75520bcc1"
Oct 31 13:04:28.869275 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:04:28.869258    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a651f0a59ca5442005a292376e368e3b7306e113cf483c2331454a75520bcc1\": container with ID starting with 4a651f0a59ca5442005a292376e368e3b7306e113cf483c2331454a75520bcc1 not found: ID does not exist" containerID="4a651f0a59ca5442005a292376e368e3b7306e113cf483c2331454a75520bcc1"
Oct 31 13:04:28.869344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.869283    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a651f0a59ca5442005a292376e368e3b7306e113cf483c2331454a75520bcc1"} err="failed to get container status \"4a651f0a59ca5442005a292376e368e3b7306e113cf483c2331454a75520bcc1\": rpc error: code = NotFound desc = could not find container \"4a651f0a59ca5442005a292376e368e3b7306e113cf483c2331454a75520bcc1\": container with ID starting with 4a651f0a59ca5442005a292376e368e3b7306e113cf483c2331454a75520bcc1 not found: ID does not exist"
Oct 31 13:04:28.881960 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.881938    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-694db45b8d-szb8q"]
Oct 31 13:04:28.885725 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:28.885702    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-694db45b8d-szb8q"]
Oct 31 13:04:30.527993 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:30.527962    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="298ad120-4782-4b13-985f-5af7fb24d97d" path="/var/lib/kubelet/pods/298ad120-4782-4b13-985f-5af7fb24d97d/volumes"
Oct 31 13:04:31.872873 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:31.872841    4660 generic.go:356] "Generic (PLEG): container finished" podID="2a1a0fc2-36f2-4635-97b6-97faed580133" containerID="e29969cb3c7add6009b36661474b30c30d52f11df4289aea516bfb37a2d4efc6" exitCode=0
Oct 31 13:04:31.873231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:31.872918    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-cpk9z" event={"ID":"2a1a0fc2-36f2-4635-97b6-97faed580133","Type":"ContainerDied","Data":"e29969cb3c7add6009b36661474b30c30d52f11df4289aea516bfb37a2d4efc6"}
Oct 31 13:04:31.873275 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:31.873244    4660 scope.go:122] "RemoveContainer" containerID="e29969cb3c7add6009b36661474b30c30d52f11df4289aea516bfb37a2d4efc6"
Oct 31 13:04:32.319312 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.319247    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-7sff7-qhcml"]
Oct 31 13:04:32.319568 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.319554    4660 cpu_manager.go:414] "containerMap: removing container" podUID="298ad120-4782-4b13-985f-5af7fb24d97d" containerName="registry"
Oct 31 13:04:32.319636 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.319570    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="298ad120-4782-4b13-985f-5af7fb24d97d" containerName="registry"
Oct 31 13:04:32.319636 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.319594    4660 cpu_manager.go:414] "containerMap: removing container" podUID="7620273c-a4f1-4e1f-ba26-453913b2e171" containerName="registry"
Oct 31 13:04:32.319636 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.319599    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7620273c-a4f1-4e1f-ba26-453913b2e171" containerName="registry"
Oct 31 13:04:32.319732 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.319675    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="7620273c-a4f1-4e1f-ba26-453913b2e171" containerName="registry"
Oct 31 13:04:32.319732 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.319690    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="298ad120-4782-4b13-985f-5af7fb24d97d" containerName="registry"
Oct 31 13:04:32.322610 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.322594    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7sff7-qhcml"
Oct 31 13:04:32.332686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.332662    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7sff7-qhcml"]
Oct 31 13:04:32.414314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.414253    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/45ff8b15-629b-481a-a3f1-595113718278-archives-path\") pod \"periodic-gathering-7sff7-qhcml\" (UID: \"45ff8b15-629b-481a-a3f1-595113718278\") " pod="openshift-insights/periodic-gathering-7sff7-qhcml"
Oct 31 13:04:32.414314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.414313    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6bd6\" (UniqueName: \"kubernetes.io/projected/45ff8b15-629b-481a-a3f1-595113718278-kube-api-access-p6bd6\") pod \"periodic-gathering-7sff7-qhcml\" (UID: \"45ff8b15-629b-481a-a3f1-595113718278\") " pod="openshift-insights/periodic-gathering-7sff7-qhcml"
Oct 31 13:04:32.414558 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.414439    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ff8b15-629b-481a-a3f1-595113718278-service-ca-bundle\") pod \"periodic-gathering-7sff7-qhcml\" (UID: \"45ff8b15-629b-481a-a3f1-595113718278\") " pod="openshift-insights/periodic-gathering-7sff7-qhcml"
Oct 31 13:04:32.414558 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.414486    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ff8b15-629b-481a-a3f1-595113718278-trusted-ca-bundle\") pod \"periodic-gathering-7sff7-qhcml\" (UID: \"45ff8b15-629b-481a-a3f1-595113718278\") " pod="openshift-insights/periodic-gathering-7sff7-qhcml"
Oct 31 13:04:32.515391 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.515357    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/45ff8b15-629b-481a-a3f1-595113718278-archives-path\") pod \"periodic-gathering-7sff7-qhcml\" (UID: \"45ff8b15-629b-481a-a3f1-595113718278\") " pod="openshift-insights/periodic-gathering-7sff7-qhcml"
Oct 31 13:04:32.515391 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.515392    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6bd6\" (UniqueName: \"kubernetes.io/projected/45ff8b15-629b-481a-a3f1-595113718278-kube-api-access-p6bd6\") pod \"periodic-gathering-7sff7-qhcml\" (UID: \"45ff8b15-629b-481a-a3f1-595113718278\") " pod="openshift-insights/periodic-gathering-7sff7-qhcml"
Oct 31 13:04:32.515644 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.515424    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ff8b15-629b-481a-a3f1-595113718278-service-ca-bundle\") pod \"periodic-gathering-7sff7-qhcml\" (UID: \"45ff8b15-629b-481a-a3f1-595113718278\") " pod="openshift-insights/periodic-gathering-7sff7-qhcml"
Oct 31 13:04:32.515644 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.515445    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ff8b15-629b-481a-a3f1-595113718278-trusted-ca-bundle\") pod \"periodic-gathering-7sff7-qhcml\" (UID: \"45ff8b15-629b-481a-a3f1-595113718278\") " pod="openshift-insights/periodic-gathering-7sff7-qhcml"
Oct 31 13:04:32.515843 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.515822    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/45ff8b15-629b-481a-a3f1-595113718278-archives-path\") pod \"periodic-gathering-7sff7-qhcml\" (UID: \"45ff8b15-629b-481a-a3f1-595113718278\") " pod="openshift-insights/periodic-gathering-7sff7-qhcml"
Oct 31 13:04:32.516070 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.516053    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ff8b15-629b-481a-a3f1-595113718278-service-ca-bundle\") pod \"periodic-gathering-7sff7-qhcml\" (UID: \"45ff8b15-629b-481a-a3f1-595113718278\") " pod="openshift-insights/periodic-gathering-7sff7-qhcml"
Oct 31 13:04:32.516164 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.516148    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ff8b15-629b-481a-a3f1-595113718278-trusted-ca-bundle\") pod \"periodic-gathering-7sff7-qhcml\" (UID: \"45ff8b15-629b-481a-a3f1-595113718278\") " pod="openshift-insights/periodic-gathering-7sff7-qhcml"
Oct 31 13:04:32.529116 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.529088    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6bd6\" (UniqueName: \"kubernetes.io/projected/45ff8b15-629b-481a-a3f1-595113718278-kube-api-access-p6bd6\") pod \"periodic-gathering-7sff7-qhcml\" (UID: \"45ff8b15-629b-481a-a3f1-595113718278\") " pod="openshift-insights/periodic-gathering-7sff7-qhcml"
Oct 31 13:04:32.633819 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.633791    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7sff7-qhcml"
Oct 31 13:04:32.877849 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.877817    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-cpk9z" event={"ID":"2a1a0fc2-36f2-4635-97b6-97faed580133","Type":"ContainerStarted","Data":"a9f2c8123bbadb81afe865746088a6cdbe21a87a9a6b6854caa832a29b062133"}
Oct 31 13:04:32.907483 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:32.907413    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7sff7-qhcml"]
Oct 31 13:04:32.910604 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:04:32.910577    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ff8b15_629b_481a_a3f1_595113718278.slice/crio-74d1b2ed3f7300da7afe8dc51cdb13e6efe12781130b58d60afdba5b44b78e67 WatchSource:0}: Error finding container 74d1b2ed3f7300da7afe8dc51cdb13e6efe12781130b58d60afdba5b44b78e67: Status 404 returned error can't find the container with id 74d1b2ed3f7300da7afe8dc51cdb13e6efe12781130b58d60afdba5b44b78e67
Oct 31 13:04:33.883342 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:33.883287    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7sff7-qhcml" event={"ID":"45ff8b15-629b-481a-a3f1-595113718278","Type":"ContainerStarted","Data":"68b2208c332eabecf47d88390c35ab5621cbfeec93257bb4bb57cd6046e32678"}
Oct 31 13:04:33.883795 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:33.883395    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7sff7-qhcml" event={"ID":"45ff8b15-629b-481a-a3f1-595113718278","Type":"ContainerStarted","Data":"74d1b2ed3f7300da7afe8dc51cdb13e6efe12781130b58d60afdba5b44b78e67"}
Oct 31 13:04:33.902249 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:33.902197    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-7sff7-qhcml" podStartSLOduration=1.9021824600000001 podStartE2EDuration="1.90218246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:04:33.901059386 +0000 UTC m=+73.519718549" watchObservedRunningTime="2026-10-31 13:04:33.90218246 +0000 UTC m=+73.520841622"
Oct 31 13:04:36.896618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:36.896537    4660 generic.go:356] "Generic (PLEG): container finished" podID="f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a" containerID="14d3d53a0676ca2730194ce8c489da3ac30b0f07c888517ec36651cbd182493d" exitCode=0
Oct 31 13:04:36.896618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:36.896580    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s" event={"ID":"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a","Type":"ContainerDied","Data":"14d3d53a0676ca2730194ce8c489da3ac30b0f07c888517ec36651cbd182493d"}
Oct 31 13:04:36.897085 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:36.896852    4660 scope.go:122] "RemoveContainer" containerID="14d3d53a0676ca2730194ce8c489da3ac30b0f07c888517ec36651cbd182493d"
Oct 31 13:04:37.901832 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:37.901802    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wwd8s" event={"ID":"f8b29c79-bfdc-4c46-9b31-c2cbf6c0a94a","Type":"ContainerStarted","Data":"067586822a2e29de8cbd3aa7c52bd812661f15382056824c9a78d22d66b9f836"}
Oct 31 13:04:41.916965 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:41.916931    4660 generic.go:356] "Generic (PLEG): container finished" podID="b3bc5c9b-54fb-477f-8c98-9884b1b2e159" containerID="42301032689b3a980e66fd0ef98156a04942ebeaa9d862a7ef596d1ae4a03fa8" exitCode=0
Oct 31 13:04:41.917357 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:41.917005    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft" event={"ID":"b3bc5c9b-54fb-477f-8c98-9884b1b2e159","Type":"ContainerDied","Data":"42301032689b3a980e66fd0ef98156a04942ebeaa9d862a7ef596d1ae4a03fa8"}
Oct 31 13:04:41.917357 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:41.917344    4660 scope.go:122] "RemoveContainer" containerID="42301032689b3a980e66fd0ef98156a04942ebeaa9d862a7ef596d1ae4a03fa8"
Oct 31 13:04:42.922207 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:04:42.922168    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-4hrft" event={"ID":"b3bc5c9b-54fb-477f-8c98-9884b1b2e159","Type":"ContainerStarted","Data":"fa828552339a2dc6a974f9aeef6bb62426585ceea5248bd73ff90b485f81566c"}
Oct 31 13:04:50.729012 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:04:50.728974    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:05:20.730119 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:05:20.730085    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:05:50.729894 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:05:50.729798    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:05:57.187915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:57.187880    4660 generic.go:356] "Generic (PLEG): container finished" podID="45ff8b15-629b-481a-a3f1-595113718278" containerID="68b2208c332eabecf47d88390c35ab5621cbfeec93257bb4bb57cd6046e32678" exitCode=0
Oct 31 13:05:57.188314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:57.187932    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7sff7-qhcml" event={"ID":"45ff8b15-629b-481a-a3f1-595113718278","Type":"ContainerDied","Data":"68b2208c332eabecf47d88390c35ab5621cbfeec93257bb4bb57cd6046e32678"}
Oct 31 13:05:58.336918 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:58.336880    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7sff7-qhcml"
Oct 31 13:05:58.490422 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:58.490332    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ff8b15-629b-481a-a3f1-595113718278-kube-api-access-p6bd6\" (UniqueName: \"kubernetes.io/projected/45ff8b15-629b-481a-a3f1-595113718278-kube-api-access-p6bd6\") pod \"45ff8b15-629b-481a-a3f1-595113718278\" (UID: \"45ff8b15-629b-481a-a3f1-595113718278\") "
Oct 31 13:05:58.490422 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:58.490379    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ff8b15-629b-481a-a3f1-595113718278-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ff8b15-629b-481a-a3f1-595113718278-service-ca-bundle\") pod \"45ff8b15-629b-481a-a3f1-595113718278\" (UID: \"45ff8b15-629b-481a-a3f1-595113718278\") "
Oct 31 13:05:58.490422 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:58.490419    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ff8b15-629b-481a-a3f1-595113718278-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ff8b15-629b-481a-a3f1-595113718278-trusted-ca-bundle\") pod \"45ff8b15-629b-481a-a3f1-595113718278\" (UID: \"45ff8b15-629b-481a-a3f1-595113718278\") "
Oct 31 13:05:58.490703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:58.490443    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45ff8b15-629b-481a-a3f1-595113718278-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/45ff8b15-629b-481a-a3f1-595113718278-archives-path\") pod \"45ff8b15-629b-481a-a3f1-595113718278\" (UID: \"45ff8b15-629b-481a-a3f1-595113718278\") "
Oct 31 13:05:58.490761 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:58.490750    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ff8b15-629b-481a-a3f1-595113718278-archives-path" pod "45ff8b15-629b-481a-a3f1-595113718278" (UID: "45ff8b15-629b-481a-a3f1-595113718278"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:05:58.490813 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:58.490754    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ff8b15-629b-481a-a3f1-595113718278-service-ca-bundle" pod "45ff8b15-629b-481a-a3f1-595113718278" (UID: "45ff8b15-629b-481a-a3f1-595113718278"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:05:58.490813 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:58.490748    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ff8b15-629b-481a-a3f1-595113718278-trusted-ca-bundle" pod "45ff8b15-629b-481a-a3f1-595113718278" (UID: "45ff8b15-629b-481a-a3f1-595113718278"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:05:58.492751 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:58.492729    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ff8b15-629b-481a-a3f1-595113718278-kube-api-access-p6bd6" pod "45ff8b15-629b-481a-a3f1-595113718278" (UID: "45ff8b15-629b-481a-a3f1-595113718278"). InnerVolumeSpecName "kube-api-access-p6bd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:05:58.591242 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:58.591201    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6bd6\" (UniqueName: \"kubernetes.io/projected/45ff8b15-629b-481a-a3f1-595113718278-kube-api-access-p6bd6\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:58.591242 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:58.591236    4660 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ff8b15-629b-481a-a3f1-595113718278-service-ca-bundle\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:58.591242 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:58.591246    4660 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ff8b15-629b-481a-a3f1-595113718278-trusted-ca-bundle\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:58.591498 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:58.591255    4660 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/45ff8b15-629b-481a-a3f1-595113718278-archives-path\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:59.199422 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:59.199385    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7sff7-qhcml" event={"ID":"45ff8b15-629b-481a-a3f1-595113718278","Type":"ContainerDied","Data":"74d1b2ed3f7300da7afe8dc51cdb13e6efe12781130b58d60afdba5b44b78e67"}
Oct 31 13:05:59.199422 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:59.199413    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7sff7-qhcml"
Oct 31 13:05:59.199620 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:05:59.199440    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74d1b2ed3f7300da7afe8dc51cdb13e6efe12781130b58d60afdba5b44b78e67"
Oct 31 13:06:20.730805 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:06:20.730765    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:06:50.729413 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:06:50.729377    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:07:20.730021 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:07:20.729979    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:07:50.729516 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:07:50.729481    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:08:20.542665 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:20.542638    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:08:20.543655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:20.543634    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:08:20.730139 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:08:20.730104    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:08:48.331691 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.331613    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c"]
Oct 31 13:08:48.332069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.331902    4660 cpu_manager.go:414] "containerMap: removing container" podUID="45ff8b15-629b-481a-a3f1-595113718278" containerName="insights-gathering"
Oct 31 13:08:48.332069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.331913    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ff8b15-629b-481a-a3f1-595113718278" containerName="insights-gathering"
Oct 31 13:08:48.332069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.331992    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ff8b15-629b-481a-a3f1-595113718278" containerName="insights-gathering"
Oct 31 13:08:48.335050 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.335032    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c"
Oct 31 13:08:48.340100 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.340072    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 13:08:48.340100 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.340095    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 13:08:48.344639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.344617    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c"]
Oct 31 13:08:48.363059 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.363039    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-ccw2w\""
Oct 31 13:08:48.374004 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.373980    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqh68\" (UniqueName: \"kubernetes.io/projected/67af19e2-81cf-4d1a-9507-454480e4953a-kube-api-access-hqh68\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c\" (UID: \"67af19e2-81cf-4d1a-9507-454480e4953a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c"
Oct 31 13:08:48.374105 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.374015    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67af19e2-81cf-4d1a-9507-454480e4953a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c\" (UID: \"67af19e2-81cf-4d1a-9507-454480e4953a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c"
Oct 31 13:08:48.374105 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.374038    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67af19e2-81cf-4d1a-9507-454480e4953a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c\" (UID: \"67af19e2-81cf-4d1a-9507-454480e4953a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c"
Oct 31 13:08:48.434835 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.434808    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d"]
Oct 31 13:08:48.438056 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.438041    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d"
Oct 31 13:08:48.442175 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.442154    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-ht8n2\""
Oct 31 13:08:48.448153 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.448131    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d"]
Oct 31 13:08:48.474902 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.474878    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqh68\" (UniqueName: \"kubernetes.io/projected/67af19e2-81cf-4d1a-9507-454480e4953a-kube-api-access-hqh68\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c\" (UID: \"67af19e2-81cf-4d1a-9507-454480e4953a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c"
Oct 31 13:08:48.475044 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.474908    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91930a15-be9e-4cd0-ad46-1b367d06e8fa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d\" (UID: \"91930a15-be9e-4cd0-ad46-1b367d06e8fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d"
Oct 31 13:08:48.475044 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.474936    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67af19e2-81cf-4d1a-9507-454480e4953a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c\" (UID: \"67af19e2-81cf-4d1a-9507-454480e4953a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c"
Oct 31 13:08:48.475159 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.475064    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91930a15-be9e-4cd0-ad46-1b367d06e8fa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d\" (UID: \"91930a15-be9e-4cd0-ad46-1b367d06e8fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d"
Oct 31 13:08:48.475159 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.475099    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67af19e2-81cf-4d1a-9507-454480e4953a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c\" (UID: \"67af19e2-81cf-4d1a-9507-454480e4953a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c"
Oct 31 13:08:48.475159 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.475127    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgjhq\" (UniqueName: \"kubernetes.io/projected/91930a15-be9e-4cd0-ad46-1b367d06e8fa-kube-api-access-fgjhq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d\" (UID: \"91930a15-be9e-4cd0-ad46-1b367d06e8fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d"
Oct 31 13:08:48.475314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.475219    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67af19e2-81cf-4d1a-9507-454480e4953a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c\" (UID: \"67af19e2-81cf-4d1a-9507-454480e4953a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c"
Oct 31 13:08:48.475406 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.475390    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67af19e2-81cf-4d1a-9507-454480e4953a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c\" (UID: \"67af19e2-81cf-4d1a-9507-454480e4953a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c"
Oct 31 13:08:48.487863 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.487843    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqh68\" (UniqueName: \"kubernetes.io/projected/67af19e2-81cf-4d1a-9507-454480e4953a-kube-api-access-hqh68\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c\" (UID: \"67af19e2-81cf-4d1a-9507-454480e4953a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c"
Oct 31 13:08:48.535891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.535863    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj"]
Oct 31 13:08:48.539199 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.539184    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj"
Oct 31 13:08:48.543308 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.543273    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-z9jn2\""
Oct 31 13:08:48.552210 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.552184    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj"]
Oct 31 13:08:48.576332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.576276    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b84c453d-748d-4287-805c-09aa3dbcced8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj\" (UID: \"b84c453d-748d-4287-805c-09aa3dbcced8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj"
Oct 31 13:08:48.576332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.576336    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b84c453d-748d-4287-805c-09aa3dbcced8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj\" (UID: \"b84c453d-748d-4287-805c-09aa3dbcced8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj"
Oct 31 13:08:48.576520 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.576448    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91930a15-be9e-4cd0-ad46-1b367d06e8fa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d\" (UID: \"91930a15-be9e-4cd0-ad46-1b367d06e8fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d"
Oct 31 13:08:48.576520 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.576495    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx6tg\" (UniqueName: \"kubernetes.io/projected/b84c453d-748d-4287-805c-09aa3dbcced8-kube-api-access-xx6tg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj\" (UID: \"b84c453d-748d-4287-805c-09aa3dbcced8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj"
Oct 31 13:08:48.576619 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.576548    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91930a15-be9e-4cd0-ad46-1b367d06e8fa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d\" (UID: \"91930a15-be9e-4cd0-ad46-1b367d06e8fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d"
Oct 31 13:08:48.576619 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.576578    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgjhq\" (UniqueName: \"kubernetes.io/projected/91930a15-be9e-4cd0-ad46-1b367d06e8fa-kube-api-access-fgjhq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d\" (UID: \"91930a15-be9e-4cd0-ad46-1b367d06e8fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d"
Oct 31 13:08:48.576778 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.576761    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91930a15-be9e-4cd0-ad46-1b367d06e8fa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d\" (UID: \"91930a15-be9e-4cd0-ad46-1b367d06e8fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d"
Oct 31 13:08:48.576829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.576812    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91930a15-be9e-4cd0-ad46-1b367d06e8fa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d\" (UID: \"91930a15-be9e-4cd0-ad46-1b367d06e8fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d"
Oct 31 13:08:48.589653 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.589605    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgjhq\" (UniqueName: \"kubernetes.io/projected/91930a15-be9e-4cd0-ad46-1b367d06e8fa-kube-api-access-fgjhq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d\" (UID: \"91930a15-be9e-4cd0-ad46-1b367d06e8fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d"
Oct 31 13:08:48.634662 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.634625    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p"]
Oct 31 13:08:48.638059 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.638038    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p"
Oct 31 13:08:48.642452 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.642431    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-tx54x\""
Oct 31 13:08:48.651021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.650998    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c"
Oct 31 13:08:48.651358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.651339    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p"]
Oct 31 13:08:48.676955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.676921    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b84c453d-748d-4287-805c-09aa3dbcced8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj\" (UID: \"b84c453d-748d-4287-805c-09aa3dbcced8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj"
Oct 31 13:08:48.677090 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.676968    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b84c453d-748d-4287-805c-09aa3dbcced8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj\" (UID: \"b84c453d-748d-4287-805c-09aa3dbcced8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj"
Oct 31 13:08:48.677090 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.677001    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl7dc\" (UniqueName: \"kubernetes.io/projected/98c5973c-b9e8-4f46-8349-dba65654eb5d-kube-api-access-fl7dc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p\" (UID: \"98c5973c-b9e8-4f46-8349-dba65654eb5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p"
Oct 31 13:08:48.677090 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.677040    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98c5973c-b9e8-4f46-8349-dba65654eb5d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p\" (UID: \"98c5973c-b9e8-4f46-8349-dba65654eb5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p"
Oct 31 13:08:48.677090 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.677075    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98c5973c-b9e8-4f46-8349-dba65654eb5d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p\" (UID: \"98c5973c-b9e8-4f46-8349-dba65654eb5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p"
Oct 31 13:08:48.677320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.677113    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx6tg\" (UniqueName: \"kubernetes.io/projected/b84c453d-748d-4287-805c-09aa3dbcced8-kube-api-access-xx6tg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj\" (UID: \"b84c453d-748d-4287-805c-09aa3dbcced8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj"
Oct 31 13:08:48.677384 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.677360    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b84c453d-748d-4287-805c-09aa3dbcced8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj\" (UID: \"b84c453d-748d-4287-805c-09aa3dbcced8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj"
Oct 31 13:08:48.677473 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.677452    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b84c453d-748d-4287-805c-09aa3dbcced8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj\" (UID: \"b84c453d-748d-4287-805c-09aa3dbcced8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj"
Oct 31 13:08:48.690636 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.690605    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx6tg\" (UniqueName: \"kubernetes.io/projected/b84c453d-748d-4287-805c-09aa3dbcced8-kube-api-access-xx6tg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj\" (UID: \"b84c453d-748d-4287-805c-09aa3dbcced8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj"
Oct 31 13:08:48.739948 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.739912    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj"]
Oct 31 13:08:48.744551 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.744532    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj"
Oct 31 13:08:48.752464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.752431    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-f46mr\""
Oct 31 13:08:48.753347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.753328    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d"
Oct 31 13:08:48.753659 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.753642    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj"]
Oct 31 13:08:48.777885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.777849    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/948938f2-e87c-4588-9803-d7886020f8ca-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj\" (UID: \"948938f2-e87c-4588-9803-d7886020f8ca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj"
Oct 31 13:08:48.778031 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.777918    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl7dc\" (UniqueName: \"kubernetes.io/projected/98c5973c-b9e8-4f46-8349-dba65654eb5d-kube-api-access-fl7dc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p\" (UID: \"98c5973c-b9e8-4f46-8349-dba65654eb5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p"
Oct 31 13:08:48.778031 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.777979    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98c5973c-b9e8-4f46-8349-dba65654eb5d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p\" (UID: \"98c5973c-b9e8-4f46-8349-dba65654eb5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p"
Oct 31 13:08:48.778031 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.778024    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/948938f2-e87c-4588-9803-d7886020f8ca-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj\" (UID: \"948938f2-e87c-4588-9803-d7886020f8ca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj"
Oct 31 13:08:48.778159 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.778042    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98c5973c-b9e8-4f46-8349-dba65654eb5d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p\" (UID: \"98c5973c-b9e8-4f46-8349-dba65654eb5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p"
Oct 31 13:08:48.778159 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.778066    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2jng\" (UniqueName: \"kubernetes.io/projected/948938f2-e87c-4588-9803-d7886020f8ca-kube-api-access-t2jng\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj\" (UID: \"948938f2-e87c-4588-9803-d7886020f8ca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj"
Oct 31 13:08:48.778433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.778411    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98c5973c-b9e8-4f46-8349-dba65654eb5d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p\" (UID: \"98c5973c-b9e8-4f46-8349-dba65654eb5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p"
Oct 31 13:08:48.778527 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.778470    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98c5973c-b9e8-4f46-8349-dba65654eb5d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p\" (UID: \"98c5973c-b9e8-4f46-8349-dba65654eb5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p"
Oct 31 13:08:48.791823 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.791797    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl7dc\" (UniqueName: \"kubernetes.io/projected/98c5973c-b9e8-4f46-8349-dba65654eb5d-kube-api-access-fl7dc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p\" (UID: \"98c5973c-b9e8-4f46-8349-dba65654eb5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p"
Oct 31 13:08:48.855656 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.855593    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj"
Oct 31 13:08:48.879001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.878964    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/948938f2-e87c-4588-9803-d7886020f8ca-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj\" (UID: \"948938f2-e87c-4588-9803-d7886020f8ca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj"
Oct 31 13:08:48.879001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.879002    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2jng\" (UniqueName: \"kubernetes.io/projected/948938f2-e87c-4588-9803-d7886020f8ca-kube-api-access-t2jng\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj\" (UID: \"948938f2-e87c-4588-9803-d7886020f8ca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj"
Oct 31 13:08:48.879210 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.879051    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/948938f2-e87c-4588-9803-d7886020f8ca-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj\" (UID: \"948938f2-e87c-4588-9803-d7886020f8ca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj"
Oct 31 13:08:48.879433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.879410    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/948938f2-e87c-4588-9803-d7886020f8ca-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj\" (UID: \"948938f2-e87c-4588-9803-d7886020f8ca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj"
Oct 31 13:08:48.879512 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.879422    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/948938f2-e87c-4588-9803-d7886020f8ca-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj\" (UID: \"948938f2-e87c-4588-9803-d7886020f8ca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj"
Oct 31 13:08:48.893436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.893409    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2jng\" (UniqueName: \"kubernetes.io/projected/948938f2-e87c-4588-9803-d7886020f8ca-kube-api-access-t2jng\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj\" (UID: \"948938f2-e87c-4588-9803-d7886020f8ca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj"
Oct 31 13:08:48.932442 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.932417    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c"]
Oct 31 13:08:48.934976 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:08:48.934952    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67af19e2_81cf_4d1a_9507_454480e4953a.slice/crio-562cc68c7d2bc63e48d7eeda3730b64f9074feaeffda2c444de745c7b2c8258b WatchSource:0}: Error finding container 562cc68c7d2bc63e48d7eeda3730b64f9074feaeffda2c444de745c7b2c8258b: Status 404 returned error can't find the container with id 562cc68c7d2bc63e48d7eeda3730b64f9074feaeffda2c444de745c7b2c8258b
Oct 31 13:08:48.937482 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.937467    4660 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:08:48.953375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:48.953350    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p"
Oct 31 13:08:49.035282 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.035255    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d"]
Oct 31 13:08:49.037459 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:08:49.037433    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91930a15_be9e_4cd0_ad46_1b367d06e8fa.slice/crio-29026baa85a7185d0cf2af50dab39ab35da9c286c500be9951a36a775c294551 WatchSource:0}: Error finding container 29026baa85a7185d0cf2af50dab39ab35da9c286c500be9951a36a775c294551: Status 404 returned error can't find the container with id 29026baa85a7185d0cf2af50dab39ab35da9c286c500be9951a36a775c294551
Oct 31 13:08:49.064592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.064569    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj"
Oct 31 13:08:49.128305 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.128259    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj"]
Oct 31 13:08:49.131873 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:08:49.131849    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84c453d_748d_4287_805c_09aa3dbcced8.slice/crio-e9c57b3aed3c8ef0e45f0bf4685e9a10f4f0f7051c8ad97276d11a95de8d53b6 WatchSource:0}: Error finding container e9c57b3aed3c8ef0e45f0bf4685e9a10f4f0f7051c8ad97276d11a95de8d53b6: Status 404 returned error can't find the container with id e9c57b3aed3c8ef0e45f0bf4685e9a10f4f0f7051c8ad97276d11a95de8d53b6
Oct 31 13:08:49.227985 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.227886    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p"]
Oct 31 13:08:49.230467 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:08:49.230437    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c5973c_b9e8_4f46_8349_dba65654eb5d.slice/crio-db2fe3a1413290e0b5550133c120978a2650a1b011b72a0b4f4b975c16474f18 WatchSource:0}: Error finding container db2fe3a1413290e0b5550133c120978a2650a1b011b72a0b4f4b975c16474f18: Status 404 returned error can't find the container with id db2fe3a1413290e0b5550133c120978a2650a1b011b72a0b4f4b975c16474f18
Oct 31 13:08:49.338459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.338423    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj"]
Oct 31 13:08:49.341371 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:08:49.341346    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948938f2_e87c_4588_9803_d7886020f8ca.slice/crio-81a4bcb2d290e03d3e369a78530531284f18e00c98b73e0c8b3358be98aa8d68 WatchSource:0}: Error finding container 81a4bcb2d290e03d3e369a78530531284f18e00c98b73e0c8b3358be98aa8d68: Status 404 returned error can't find the container with id 81a4bcb2d290e03d3e369a78530531284f18e00c98b73e0c8b3358be98aa8d68
Oct 31 13:08:49.356274 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.356249    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl"]
Oct 31 13:08:49.359692 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.359675    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl"
Oct 31 13:08:49.364122 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.364078    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-rxmvd\""
Oct 31 13:08:49.370203 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.370178    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl"]
Oct 31 13:08:49.383437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.383406    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzq9r\" (UniqueName: \"kubernetes.io/projected/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-kube-api-access-tzq9r\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl\" (UID: \"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl"
Oct 31 13:08:49.383529 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.383460    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl\" (UID: \"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl"
Oct 31 13:08:49.383529 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.383495    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl\" (UID: \"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl"
Oct 31 13:08:49.484846 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.484808    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzq9r\" (UniqueName: \"kubernetes.io/projected/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-kube-api-access-tzq9r\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl\" (UID: \"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl"
Oct 31 13:08:49.485008 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.484860    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl\" (UID: \"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl"
Oct 31 13:08:49.485008 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.484920    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl\" (UID: \"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl"
Oct 31 13:08:49.485255 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.485232    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl\" (UID: \"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl"
Oct 31 13:08:49.485355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.485270    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl\" (UID: \"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl"
Oct 31 13:08:49.498940 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.498918    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzq9r\" (UniqueName: \"kubernetes.io/projected/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-kube-api-access-tzq9r\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl\" (UID: \"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl"
Oct 31 13:08:49.678763 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.678724    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl"
Oct 31 13:08:49.780750 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.780006    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p" event={"ID":"98c5973c-b9e8-4f46-8349-dba65654eb5d","Type":"ContainerStarted","Data":"db2fe3a1413290e0b5550133c120978a2650a1b011b72a0b4f4b975c16474f18"}
Oct 31 13:08:49.785942 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.781893    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj" event={"ID":"b84c453d-748d-4287-805c-09aa3dbcced8","Type":"ContainerStarted","Data":"e9c57b3aed3c8ef0e45f0bf4685e9a10f4f0f7051c8ad97276d11a95de8d53b6"}
Oct 31 13:08:49.785942 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.783538    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c" event={"ID":"67af19e2-81cf-4d1a-9507-454480e4953a","Type":"ContainerStarted","Data":"562cc68c7d2bc63e48d7eeda3730b64f9074feaeffda2c444de745c7b2c8258b"}
Oct 31 13:08:49.785942 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.784940    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj" event={"ID":"948938f2-e87c-4588-9803-d7886020f8ca","Type":"ContainerStarted","Data":"81a4bcb2d290e03d3e369a78530531284f18e00c98b73e0c8b3358be98aa8d68"}
Oct 31 13:08:49.790358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.788781    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d" event={"ID":"91930a15-be9e-4cd0-ad46-1b367d06e8fa","Type":"ContainerStarted","Data":"29026baa85a7185d0cf2af50dab39ab35da9c286c500be9951a36a775c294551"}
Oct 31 13:08:49.986381 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:49.985223    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl"]
Oct 31 13:08:49.992939 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:08:49.992905    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8494cb1_cd4f_494e_bb4a_4ad0cb406e79.slice/crio-f1b71843b3585ee1fdd7e1f1285428670b147658f7cb12da0b553798b69f2a7a WatchSource:0}: Error finding container f1b71843b3585ee1fdd7e1f1285428670b147658f7cb12da0b553798b69f2a7a: Status 404 returned error can't find the container with id f1b71843b3585ee1fdd7e1f1285428670b147658f7cb12da0b553798b69f2a7a
Oct 31 13:08:50.737085 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:08:50.737049    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:08:50.795425 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:50.795385    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl" event={"ID":"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79","Type":"ContainerStarted","Data":"f1b71843b3585ee1fdd7e1f1285428670b147658f7cb12da0b553798b69f2a7a"}
Oct 31 13:08:52.567050 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.567014    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q"]
Oct 31 13:08:52.571212 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.571185    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q"
Oct 31 13:08:52.576720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.576688    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-2m2lx\""
Oct 31 13:08:52.582053 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.581320    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q"]
Oct 31 13:08:52.610980 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.610936    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q\" (UID: \"dbce6e87-58e5-4f07-bdd2-5f2eb402836a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q"
Oct 31 13:08:52.611168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.611027    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtpgp\" (UniqueName: \"kubernetes.io/projected/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-kube-api-access-dtpgp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q\" (UID: \"dbce6e87-58e5-4f07-bdd2-5f2eb402836a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q"
Oct 31 13:08:52.611168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.611157    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q\" (UID: \"dbce6e87-58e5-4f07-bdd2-5f2eb402836a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q"
Oct 31 13:08:52.711835 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.711796    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q\" (UID: \"dbce6e87-58e5-4f07-bdd2-5f2eb402836a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q"
Oct 31 13:08:52.712029 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.711868    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q\" (UID: \"dbce6e87-58e5-4f07-bdd2-5f2eb402836a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q"
Oct 31 13:08:52.712029 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.711904    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtpgp\" (UniqueName: \"kubernetes.io/projected/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-kube-api-access-dtpgp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q\" (UID: \"dbce6e87-58e5-4f07-bdd2-5f2eb402836a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q"
Oct 31 13:08:52.712227 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.712204    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q\" (UID: \"dbce6e87-58e5-4f07-bdd2-5f2eb402836a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q"
Oct 31 13:08:52.712482 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.712463    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q\" (UID: \"dbce6e87-58e5-4f07-bdd2-5f2eb402836a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q"
Oct 31 13:08:52.726490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.726442    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"]
Oct 31 13:08:52.728149 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.728089    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtpgp\" (UniqueName: \"kubernetes.io/projected/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-kube-api-access-dtpgp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q\" (UID: \"dbce6e87-58e5-4f07-bdd2-5f2eb402836a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q"
Oct 31 13:08:52.731085 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.730485    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.735021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.735000    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 13:08:52.743075 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.743043    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"]
Oct 31 13:08:52.765241 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.765174    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-r8f7r\""
Oct 31 13:08:52.814226 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.813588    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.814226 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.813739    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2225\" (UniqueName: \"kubernetes.io/projected/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-kube-api-access-w2225\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.814558 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.814534    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.814655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.814584    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.814718 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.814703    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.814777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.814734    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.814777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.814767    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.814881 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.814788    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.814881 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.814826    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.816850 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.816824    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"]
Oct 31 13:08:52.821840 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.821778    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:52.833629 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.831897    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-ps5xk\""
Oct 31 13:08:52.833629 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.833560    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"]
Oct 31 13:08:52.893811 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.893770    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q"
Oct 31 13:08:52.916002 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.915961    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x649z\" (UniqueName: \"kubernetes.io/projected/26126910-4393-4b5e-b788-71db446afb82-kube-api-access-x649z\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:52.916106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916030    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.916106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916061    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2225\" (UniqueName: \"kubernetes.io/projected/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-kube-api-access-w2225\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.916106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916099    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26126910-4393-4b5e-b788-71db446afb82-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:52.916220 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916131    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.916220 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916162    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26126910-4393-4b5e-b788-71db446afb82-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:52.916220 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916186    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.916353 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916215    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26126910-4393-4b5e-b788-71db446afb82-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:52.916406 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916340    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26126910-4393-4b5e-b788-71db446afb82-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:52.916450 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916424    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26126910-4393-4b5e-b788-71db446afb82-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:52.916494 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916474    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26126910-4393-4b5e-b788-71db446afb82-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:52.916529 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916507    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26126910-4393-4b5e-b788-71db446afb82-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:52.916563 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916536    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.916596 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916566    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.916628 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916608    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.916658 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916633    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26126910-4393-4b5e-b788-71db446afb82-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:52.916690 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916667    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.916726 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.916710    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.917734 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.917707    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.917833 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.917759    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.918006 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.917979    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.918169 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.918144    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.918497 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.918468    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.932085 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.932046    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.936444 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.936410    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.936598 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.936439    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:52.937045 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:52.937022    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2225\" (UniqueName: \"kubernetes.io/projected/ab9e59d3-2ac8-4381-98fa-22d3f25d682b-kube-api-access-w2225\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-2fpt9\" (UID: \"ab9e59d3-2ac8-4381-98fa-22d3f25d682b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:53.017479 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.017418    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26126910-4393-4b5e-b788-71db446afb82-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.017653 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.017553    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x649z\" (UniqueName: \"kubernetes.io/projected/26126910-4393-4b5e-b788-71db446afb82-kube-api-access-x649z\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.017725 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.017700    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26126910-4393-4b5e-b788-71db446afb82-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.017859 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.017839    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26126910-4393-4b5e-b788-71db446afb82-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.017933 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.017888    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26126910-4393-4b5e-b788-71db446afb82-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.017933 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.017913    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26126910-4393-4b5e-b788-71db446afb82-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.018038 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.017946    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26126910-4393-4b5e-b788-71db446afb82-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.018038 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.017994    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26126910-4393-4b5e-b788-71db446afb82-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.018038 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.018023    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26126910-4393-4b5e-b788-71db446afb82-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.019598 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.019520    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26126910-4393-4b5e-b788-71db446afb82-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.020323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.019813    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26126910-4393-4b5e-b788-71db446afb82-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.020323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.020067    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26126910-4393-4b5e-b788-71db446afb82-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.020323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.020111    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26126910-4393-4b5e-b788-71db446afb82-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.020500 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.020340    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26126910-4393-4b5e-b788-71db446afb82-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.022241 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.021735    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26126910-4393-4b5e-b788-71db446afb82-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.022241 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.022185    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26126910-4393-4b5e-b788-71db446afb82-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.032771 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.032687    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x649z\" (UniqueName: \"kubernetes.io/projected/26126910-4393-4b5e-b788-71db446afb82-kube-api-access-x649z\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.033010 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.032883    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26126910-4393-4b5e-b788-71db446afb82-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6\" (UID: \"26126910-4393-4b5e-b788-71db446afb82\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.060745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.060705    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:08:53.139983 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.139947    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:08:53.200738 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.200704    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q"]
Oct 31 13:08:53.206027 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:08:53.205993    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbce6e87_58e5_4f07_bdd2_5f2eb402836a.slice/crio-dda6f6cf76da37c557250722715c8e4b75f5dd570ee7ab48ee7465de00954612 WatchSource:0}: Error finding container dda6f6cf76da37c557250722715c8e4b75f5dd570ee7ab48ee7465de00954612: Status 404 returned error can't find the container with id dda6f6cf76da37c557250722715c8e4b75f5dd570ee7ab48ee7465de00954612
Oct 31 13:08:53.443411 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.442679    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"]
Oct 31 13:08:53.451869 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:08:53.451838    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9e59d3_2ac8_4381_98fa_22d3f25d682b.slice/crio-f9827b1dcbdbff170dfbc352f16e264efc9973c47f1e24d9f0d7cabb0b181932 WatchSource:0}: Error finding container f9827b1dcbdbff170dfbc352f16e264efc9973c47f1e24d9f0d7cabb0b181932: Status 404 returned error can't find the container with id f9827b1dcbdbff170dfbc352f16e264efc9973c47f1e24d9f0d7cabb0b181932
Oct 31 13:08:53.484460 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.484336    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"]
Oct 31 13:08:53.818260 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.818122    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9" event={"ID":"ab9e59d3-2ac8-4381-98fa-22d3f25d682b","Type":"ContainerStarted","Data":"f9827b1dcbdbff170dfbc352f16e264efc9973c47f1e24d9f0d7cabb0b181932"}
Oct 31 13:08:53.820213 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.820157    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q" event={"ID":"dbce6e87-58e5-4f07-bdd2-5f2eb402836a","Type":"ContainerStarted","Data":"dda6f6cf76da37c557250722715c8e4b75f5dd570ee7ab48ee7465de00954612"}
Oct 31 13:08:53.821813 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:53.821781    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6" event={"ID":"26126910-4393-4b5e-b788-71db446afb82","Type":"ContainerStarted","Data":"1606171b1e67e43cc1c00f262e22b9e118fa065d8670f84874ed0db44117a666"}
Oct 31 13:08:55.213920 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.213890    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf"]
Oct 31 13:08:55.219689 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.219666    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf"
Oct 31 13:08:55.232998 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.232553    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf"]
Oct 31 13:08:55.247045 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.247011    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49"]
Oct 31 13:08:55.253716 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.253220    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49"
Oct 31 13:08:55.291539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.291508    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49"]
Oct 31 13:08:55.298375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.298339    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4"]
Oct 31 13:08:55.334238 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.333191    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4"]
Oct 31 13:08:55.334238 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.333273    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4"
Oct 31 13:08:55.334238 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.333283    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf"]
Oct 31 13:08:55.344254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.344231    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf"]
Oct 31 13:08:55.344420 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.344375    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf"
Oct 31 13:08:55.352987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.352907    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/986b9082-b05e-4fda-b543-40b5931c2df0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf\" (UID: \"986b9082-b05e-4fda-b543-40b5931c2df0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf"
Oct 31 13:08:55.353156 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.353015    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49\" (UID: \"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49"
Oct 31 13:08:55.353156 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.353059    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9pmf\" (UniqueName: \"kubernetes.io/projected/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-kube-api-access-k9pmf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49\" (UID: \"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49"
Oct 31 13:08:55.353156 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.353076    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbvwg\" (UniqueName: \"kubernetes.io/projected/986b9082-b05e-4fda-b543-40b5931c2df0-kube-api-access-vbvwg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf\" (UID: \"986b9082-b05e-4fda-b543-40b5931c2df0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf"
Oct 31 13:08:55.353156 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.353118    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49\" (UID: \"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49"
Oct 31 13:08:55.353156 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.353135    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/986b9082-b05e-4fda-b543-40b5931c2df0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf\" (UID: \"986b9082-b05e-4fda-b543-40b5931c2df0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf"
Oct 31 13:08:55.454185 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.454152    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49\" (UID: \"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49"
Oct 31 13:08:55.454185 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.454195    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/986b9082-b05e-4fda-b543-40b5931c2df0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf\" (UID: \"986b9082-b05e-4fda-b543-40b5931c2df0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf"
Oct 31 13:08:55.456623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.454227    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/345cc59b-0448-4ecb-ac99-2d9b68c1744b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4\" (UID: \"345cc59b-0448-4ecb-ac99-2d9b68c1744b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4"
Oct 31 13:08:55.456623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.454305    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/986b9082-b05e-4fda-b543-40b5931c2df0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf\" (UID: \"986b9082-b05e-4fda-b543-40b5931c2df0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf"
Oct 31 13:08:55.456623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.454714    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49\" (UID: \"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49"
Oct 31 13:08:55.456623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.454794    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/345cc59b-0448-4ecb-ac99-2d9b68c1744b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4\" (UID: \"345cc59b-0448-4ecb-ac99-2d9b68c1744b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4"
Oct 31 13:08:55.456623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.454839    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49\" (UID: \"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49"
Oct 31 13:08:55.456623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.454866    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5e713c6-941d-4219-afde-75e3d40feb51-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf\" (UID: \"b5e713c6-941d-4219-afde-75e3d40feb51\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf"
Oct 31 13:08:55.456623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.454931    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9pmf\" (UniqueName: \"kubernetes.io/projected/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-kube-api-access-k9pmf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49\" (UID: \"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49"
Oct 31 13:08:55.456623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.454960    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbvwg\" (UniqueName: \"kubernetes.io/projected/986b9082-b05e-4fda-b543-40b5931c2df0-kube-api-access-vbvwg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf\" (UID: \"986b9082-b05e-4fda-b543-40b5931c2df0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf"
Oct 31 13:08:55.456623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.454997    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5e713c6-941d-4219-afde-75e3d40feb51-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf\" (UID: \"b5e713c6-941d-4219-afde-75e3d40feb51\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf"
Oct 31 13:08:55.456623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.455025    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hp6m\" (UniqueName: \"kubernetes.io/projected/345cc59b-0448-4ecb-ac99-2d9b68c1744b-kube-api-access-6hp6m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4\" (UID: \"345cc59b-0448-4ecb-ac99-2d9b68c1744b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4"
Oct 31 13:08:55.456623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.455051    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9jqm\" (UniqueName: \"kubernetes.io/projected/b5e713c6-941d-4219-afde-75e3d40feb51-kube-api-access-n9jqm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf\" (UID: \"b5e713c6-941d-4219-afde-75e3d40feb51\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf"
Oct 31 13:08:55.456623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.455399    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49\" (UID: \"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49"
Oct 31 13:08:55.456623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.456322    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/986b9082-b05e-4fda-b543-40b5931c2df0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf\" (UID: \"986b9082-b05e-4fda-b543-40b5931c2df0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf"
Oct 31 13:08:55.457656 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.457617    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/986b9082-b05e-4fda-b543-40b5931c2df0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf\" (UID: \"986b9082-b05e-4fda-b543-40b5931c2df0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf"
Oct 31 13:08:55.477638 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.477544    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbvwg\" (UniqueName: \"kubernetes.io/projected/986b9082-b05e-4fda-b543-40b5931c2df0-kube-api-access-vbvwg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf\" (UID: \"986b9082-b05e-4fda-b543-40b5931c2df0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf"
Oct 31 13:08:55.479194 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.479158    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9pmf\" (UniqueName: \"kubernetes.io/projected/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-kube-api-access-k9pmf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49\" (UID: \"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49"
Oct 31 13:08:55.543247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.542971    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf"
Oct 31 13:08:55.558054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.555990    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5e713c6-941d-4219-afde-75e3d40feb51-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf\" (UID: \"b5e713c6-941d-4219-afde-75e3d40feb51\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf"
Oct 31 13:08:55.558054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.556043    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hp6m\" (UniqueName: \"kubernetes.io/projected/345cc59b-0448-4ecb-ac99-2d9b68c1744b-kube-api-access-6hp6m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4\" (UID: \"345cc59b-0448-4ecb-ac99-2d9b68c1744b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4"
Oct 31 13:08:55.558054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.556074    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9jqm\" (UniqueName: \"kubernetes.io/projected/b5e713c6-941d-4219-afde-75e3d40feb51-kube-api-access-n9jqm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf\" (UID: \"b5e713c6-941d-4219-afde-75e3d40feb51\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf"
Oct 31 13:08:55.558054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.556112    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/345cc59b-0448-4ecb-ac99-2d9b68c1744b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4\" (UID: \"345cc59b-0448-4ecb-ac99-2d9b68c1744b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4"
Oct 31 13:08:55.558054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.556188    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/345cc59b-0448-4ecb-ac99-2d9b68c1744b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4\" (UID: \"345cc59b-0448-4ecb-ac99-2d9b68c1744b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4"
Oct 31 13:08:55.558054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.556226    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5e713c6-941d-4219-afde-75e3d40feb51-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf\" (UID: \"b5e713c6-941d-4219-afde-75e3d40feb51\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf"
Oct 31 13:08:55.558054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.556685    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5e713c6-941d-4219-afde-75e3d40feb51-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf\" (UID: \"b5e713c6-941d-4219-afde-75e3d40feb51\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf"
Oct 31 13:08:55.558054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.556913    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5e713c6-941d-4219-afde-75e3d40feb51-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf\" (UID: \"b5e713c6-941d-4219-afde-75e3d40feb51\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf"
Oct 31 13:08:55.558054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.557778    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/345cc59b-0448-4ecb-ac99-2d9b68c1744b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4\" (UID: \"345cc59b-0448-4ecb-ac99-2d9b68c1744b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4"
Oct 31 13:08:55.558054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.558002    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/345cc59b-0448-4ecb-ac99-2d9b68c1744b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4\" (UID: \"345cc59b-0448-4ecb-ac99-2d9b68c1744b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4"
Oct 31 13:08:55.571959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.571923    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9jqm\" (UniqueName: \"kubernetes.io/projected/b5e713c6-941d-4219-afde-75e3d40feb51-kube-api-access-n9jqm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf\" (UID: \"b5e713c6-941d-4219-afde-75e3d40feb51\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf"
Oct 31 13:08:55.574140 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.574096    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hp6m\" (UniqueName: \"kubernetes.io/projected/345cc59b-0448-4ecb-ac99-2d9b68c1744b-kube-api-access-6hp6m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4\" (UID: \"345cc59b-0448-4ecb-ac99-2d9b68c1744b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4"
Oct 31 13:08:55.599974 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.599699    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49"
Oct 31 13:08:55.674408 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.674343    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4"
Oct 31 13:08:55.688730 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.688437    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf"
Oct 31 13:08:55.908866 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.908544    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf"]
Oct 31 13:08:55.914683 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:08:55.914623    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986b9082_b05e_4fda_b543_40b5931c2df0.slice/crio-f2b9dd5ce118561e6283b0d3104f56c86008d0288d05435e4cd34ff0de6885ee WatchSource:0}: Error finding container f2b9dd5ce118561e6283b0d3104f56c86008d0288d05435e4cd34ff0de6885ee: Status 404 returned error can't find the container with id f2b9dd5ce118561e6283b0d3104f56c86008d0288d05435e4cd34ff0de6885ee
Oct 31 13:08:55.929251 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:55.929214    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49"]
Oct 31 13:08:56.026376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:56.026276    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4"]
Oct 31 13:08:56.050606 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:08:56.050557    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345cc59b_0448_4ecb_ac99_2d9b68c1744b.slice/crio-01f41099b7ce749aa48da478b9125531d5ebd22d26f52aa758da7d179782b002 WatchSource:0}: Error finding container 01f41099b7ce749aa48da478b9125531d5ebd22d26f52aa758da7d179782b002: Status 404 returned error can't find the container with id 01f41099b7ce749aa48da478b9125531d5ebd22d26f52aa758da7d179782b002
Oct 31 13:08:56.072212 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:56.071523    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf"]
Oct 31 13:08:56.872608 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:56.872567    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4" event={"ID":"345cc59b-0448-4ecb-ac99-2d9b68c1744b","Type":"ContainerStarted","Data":"01f41099b7ce749aa48da478b9125531d5ebd22d26f52aa758da7d179782b002"}
Oct 31 13:08:56.900002 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:56.899919    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf" event={"ID":"986b9082-b05e-4fda-b543-40b5931c2df0","Type":"ContainerStarted","Data":"f2b9dd5ce118561e6283b0d3104f56c86008d0288d05435e4cd34ff0de6885ee"}
Oct 31 13:08:56.910524 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:56.910482    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf" event={"ID":"b5e713c6-941d-4219-afde-75e3d40feb51","Type":"ContainerStarted","Data":"3d4477fc347c7f22acdd776eeac17da6024e0ea22570acc9be97576a23c382e4"}
Oct 31 13:08:56.914231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:08:56.914191    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49" event={"ID":"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d","Type":"ContainerStarted","Data":"3089bccc3f9daf51556723400d6876c60649e2728b2ae17319fa4be663a78731"}
Oct 31 13:09:06.514733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:06.514693    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:09:06.515132 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:06.514786    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:09:06.515132 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:06.514832    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:09:06.538519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:06.537613    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:09:06.538519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:06.537711    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:09:06.538519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:06.537753    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:09:07.062354 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:07.062245    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9" event={"ID":"ab9e59d3-2ac8-4381-98fa-22d3f25d682b","Type":"ContainerStarted","Data":"7f75e2cdf47ba65a60221b84d475060213db568023350d096f246c98205203fe"}
Oct 31 13:09:07.081827 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:07.081783    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6" event={"ID":"26126910-4393-4b5e-b788-71db446afb82","Type":"ContainerStarted","Data":"d40e2090091e74177357b5469116fabfed19b18657cbc6e7ebaba2a81e04db56"}
Oct 31 13:09:07.093568 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:07.092251    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9" podStartSLOduration=2.012955151 podStartE2EDuration="15.092233126s" totalImagesPullingTime="13.079277975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:07.08936009 +0000 UTC m=+346.708019257" watchObservedRunningTime="2026-10-31 13:09:07.092233126 +0000 UTC m=+346.710892291"
Oct 31 13:09:07.141411 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:07.141373    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:09:07.148119 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:07.148087    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:09:07.176899 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:07.174573    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6" podStartSLOduration=2.153858944 podStartE2EDuration="15.174552542s" totalImagesPullingTime="13.020693598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:07.120238184 +0000 UTC m=+346.738897346" watchObservedRunningTime="2026-10-31 13:09:07.174552542 +0000 UTC m=+346.793211708"
Oct 31 13:09:08.061941 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:08.061469    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:09:08.061941 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:08.061828    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:09:08.069239 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:08.068953    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:09:08.093323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:08.092224    4660 generic.go:356] "Generic (PLEG): container finished" podID="91930a15-be9e-4cd0-ad46-1b367d06e8fa" containerID="4865565d5d420bbe04cd310dc46159a53a3a36119ec91dc8bb018c07cdef19b8" exitCode=0
Oct 31 13:09:08.093323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:08.093025    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d" event={"ID":"91930a15-be9e-4cd0-ad46-1b367d06e8fa","Type":"ContainerDied","Data":"4865565d5d420bbe04cd310dc46159a53a3a36119ec91dc8bb018c07cdef19b8"}
Oct 31 13:09:08.093323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:08.093075    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:09:08.096619 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:08.095005    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6"
Oct 31 13:09:08.097402 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:08.097353    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:09:09.106130 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:09.104906    4660 generic.go:356] "Generic (PLEG): container finished" podID="f8494cb1-cd4f-494e-bb4a-4ad0cb406e79" containerID="fc46299fdf5eeb88292397c8693450d8d4228b301801878617cbfefe3a57f921" exitCode=0
Oct 31 13:09:09.106130 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:09.105355    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl" event={"ID":"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79","Type":"ContainerDied","Data":"fc46299fdf5eeb88292397c8693450d8d4228b301801878617cbfefe3a57f921"}
Oct 31 13:09:10.114200 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:10.114160    4660 generic.go:356] "Generic (PLEG): container finished" podID="345cc59b-0448-4ecb-ac99-2d9b68c1744b" containerID="8d5066b0519798cc47337b70d571f16e25b7a41fa19e9256619643a71fbed62f" exitCode=0
Oct 31 13:09:10.114703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:10.114282    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4" event={"ID":"345cc59b-0448-4ecb-ac99-2d9b68c1744b","Type":"ContainerDied","Data":"8d5066b0519798cc47337b70d571f16e25b7a41fa19e9256619643a71fbed62f"}
Oct 31 13:09:10.119266 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:10.119236    4660 generic.go:356] "Generic (PLEG): container finished" podID="b5e713c6-941d-4219-afde-75e3d40feb51" containerID="55513a994b52ba628a44ca571ae7acdba93c8446020d40ffcbcb3b2683acff34" exitCode=0
Oct 31 13:09:10.119389 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:10.119343    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf" event={"ID":"b5e713c6-941d-4219-afde-75e3d40feb51","Type":"ContainerDied","Data":"55513a994b52ba628a44ca571ae7acdba93c8446020d40ffcbcb3b2683acff34"}
Oct 31 13:09:10.127274 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:10.122939    4660 generic.go:356] "Generic (PLEG): container finished" podID="2e19280d-a86c-48cd-b4e1-7cbe8308cb7d" containerID="05637dd407d3c7e53611a20ef08130e1b2c34b77de7a7de69f3939ca5d9d7022" exitCode=0
Oct 31 13:09:10.127274 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:10.124132    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49" event={"ID":"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d","Type":"ContainerDied","Data":"05637dd407d3c7e53611a20ef08130e1b2c34b77de7a7de69f3939ca5d9d7022"}
Oct 31 13:09:11.138030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:11.136947    4660 generic.go:356] "Generic (PLEG): container finished" podID="986b9082-b05e-4fda-b543-40b5931c2df0" containerID="9467d1e1746c9701455dd0d895814830620c805866eac7bbe8ee446e08aa6816" exitCode=0
Oct 31 13:09:11.138030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:11.137107    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf" event={"ID":"986b9082-b05e-4fda-b543-40b5931c2df0","Type":"ContainerDied","Data":"9467d1e1746c9701455dd0d895814830620c805866eac7bbe8ee446e08aa6816"}
Oct 31 13:09:11.142978 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:11.142866    4660 generic.go:356] "Generic (PLEG): container finished" podID="91930a15-be9e-4cd0-ad46-1b367d06e8fa" containerID="fa05ce3fa664a3447dd9d9be7adfaf51ec948cbd579d0dd0e54d4d1f7a271b21" exitCode=0
Oct 31 13:09:11.142978 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:11.142923    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d" event={"ID":"91930a15-be9e-4cd0-ad46-1b367d06e8fa","Type":"ContainerDied","Data":"fa05ce3fa664a3447dd9d9be7adfaf51ec948cbd579d0dd0e54d4d1f7a271b21"}
Oct 31 13:09:12.163210 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:12.163127    4660 generic.go:356] "Generic (PLEG): container finished" podID="67af19e2-81cf-4d1a-9507-454480e4953a" containerID="132e700fc5546ff3f436dd22e4c221c72a433e5abe6e5724faee2fc0bd049756" exitCode=0
Oct 31 13:09:12.163725 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:12.163215    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c" event={"ID":"67af19e2-81cf-4d1a-9507-454480e4953a","Type":"ContainerDied","Data":"132e700fc5546ff3f436dd22e4c221c72a433e5abe6e5724faee2fc0bd049756"}
Oct 31 13:09:12.170889 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:12.170850    4660 generic.go:356] "Generic (PLEG): container finished" podID="345cc59b-0448-4ecb-ac99-2d9b68c1744b" containerID="a518be1e9c3ce911809a6197ccbfbef7670ce888692814e9b941b2d5af594bdc" exitCode=0
Oct 31 13:09:12.170989 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:12.170950    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4" event={"ID":"345cc59b-0448-4ecb-ac99-2d9b68c1744b","Type":"ContainerDied","Data":"a518be1e9c3ce911809a6197ccbfbef7670ce888692814e9b941b2d5af594bdc"}
Oct 31 13:09:13.178775 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:13.178599    4660 generic.go:356] "Generic (PLEG): container finished" podID="948938f2-e87c-4588-9803-d7886020f8ca" containerID="91b3e51c6729b15f70182ee2a17cdbc7df7267b1153528d06e8e9ed1b8eaca51" exitCode=0
Oct 31 13:09:13.178775 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:13.178693    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj" event={"ID":"948938f2-e87c-4588-9803-d7886020f8ca","Type":"ContainerDied","Data":"91b3e51c6729b15f70182ee2a17cdbc7df7267b1153528d06e8e9ed1b8eaca51"}
Oct 31 13:09:13.184304 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:13.181898    4660 generic.go:356] "Generic (PLEG): container finished" podID="b5e713c6-941d-4219-afde-75e3d40feb51" containerID="691c06e41929b6409dc985bb6bbb9fb9af3d5101c0108a6d58efc327014b1bcc" exitCode=0
Oct 31 13:09:13.184304 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:13.182037    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf" event={"ID":"b5e713c6-941d-4219-afde-75e3d40feb51","Type":"ContainerDied","Data":"691c06e41929b6409dc985bb6bbb9fb9af3d5101c0108a6d58efc327014b1bcc"}
Oct 31 13:09:13.185942 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:13.185302    4660 generic.go:356] "Generic (PLEG): container finished" podID="98c5973c-b9e8-4f46-8349-dba65654eb5d" containerID="494bd0da9eb593f8d1a18df278bbfaa68fea19656108f67c3d971486f271eb1d" exitCode=0
Oct 31 13:09:13.185942 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:13.185379    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p" event={"ID":"98c5973c-b9e8-4f46-8349-dba65654eb5d","Type":"ContainerDied","Data":"494bd0da9eb593f8d1a18df278bbfaa68fea19656108f67c3d971486f271eb1d"}
Oct 31 13:09:13.187974 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:13.187945    4660 generic.go:356] "Generic (PLEG): container finished" podID="b84c453d-748d-4287-805c-09aa3dbcced8" containerID="7b0f910885afbd91525c410931b2a43c59d47038f6244557db2bdc1510f1ffcb" exitCode=0
Oct 31 13:09:13.188579 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:13.188536    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj" event={"ID":"b84c453d-748d-4287-805c-09aa3dbcced8","Type":"ContainerDied","Data":"7b0f910885afbd91525c410931b2a43c59d47038f6244557db2bdc1510f1ffcb"}
Oct 31 13:09:13.194646 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:13.194616    4660 generic.go:356] "Generic (PLEG): container finished" podID="67af19e2-81cf-4d1a-9507-454480e4953a" containerID="520bdb3cb9744c67bb39530e166bd0e3cf24988f4aabd1655bbc59efd701766f" exitCode=0
Oct 31 13:09:13.194740 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:13.194694    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c" event={"ID":"67af19e2-81cf-4d1a-9507-454480e4953a","Type":"ContainerDied","Data":"520bdb3cb9744c67bb39530e166bd0e3cf24988f4aabd1655bbc59efd701766f"}
Oct 31 13:09:13.196775 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:13.196525    4660 generic.go:356] "Generic (PLEG): container finished" podID="dbce6e87-58e5-4f07-bdd2-5f2eb402836a" containerID="7dba6e8b2ec9459cef3f11b30b17d0247c3d731e17433043306c815a7d2d436c" exitCode=0
Oct 31 13:09:13.196775 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:13.196611    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q" event={"ID":"dbce6e87-58e5-4f07-bdd2-5f2eb402836a","Type":"ContainerDied","Data":"7dba6e8b2ec9459cef3f11b30b17d0247c3d731e17433043306c815a7d2d436c"}
Oct 31 13:09:13.204264 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:13.204163    4660 generic.go:356] "Generic (PLEG): container finished" podID="986b9082-b05e-4fda-b543-40b5931c2df0" containerID="d29afd540d6a9ce833df9ad18d4603b1b7434b3ee1e83a21d7d61bf6d339a4dc" exitCode=0
Oct 31 13:09:13.204264 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:13.204196    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf" event={"ID":"986b9082-b05e-4fda-b543-40b5931c2df0","Type":"ContainerDied","Data":"d29afd540d6a9ce833df9ad18d4603b1b7434b3ee1e83a21d7d61bf6d339a4dc"}
Oct 31 13:09:14.214909 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:14.214869    4660 generic.go:356] "Generic (PLEG): container finished" podID="948938f2-e87c-4588-9803-d7886020f8ca" containerID="65032692e06b116dc0106ecbfd0d82b593eb18e78d273799b21e48b5714ca085" exitCode=0
Oct 31 13:09:14.215518 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:14.214962    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj" event={"ID":"948938f2-e87c-4588-9803-d7886020f8ca","Type":"ContainerDied","Data":"65032692e06b116dc0106ecbfd0d82b593eb18e78d273799b21e48b5714ca085"}
Oct 31 13:09:14.218541 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:14.218510    4660 generic.go:356] "Generic (PLEG): container finished" podID="f8494cb1-cd4f-494e-bb4a-4ad0cb406e79" containerID="d743240bf3a69ff1de93d5566592bbe5694706c870b1b28172022fb5f98dd8dd" exitCode=0
Oct 31 13:09:14.218637 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:14.218594    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl" event={"ID":"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79","Type":"ContainerDied","Data":"d743240bf3a69ff1de93d5566592bbe5694706c870b1b28172022fb5f98dd8dd"}
Oct 31 13:09:14.225738 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:14.225128    4660 generic.go:356] "Generic (PLEG): container finished" podID="2e19280d-a86c-48cd-b4e1-7cbe8308cb7d" containerID="0e14a7b8edd74e82854766cec3de69dcefa44d9e6b10b9a85284aa3222c3d396" exitCode=0
Oct 31 13:09:14.225738 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:14.225248    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49" event={"ID":"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d","Type":"ContainerDied","Data":"0e14a7b8edd74e82854766cec3de69dcefa44d9e6b10b9a85284aa3222c3d396"}
Oct 31 13:09:14.229548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:14.229527    4660 generic.go:356] "Generic (PLEG): container finished" podID="98c5973c-b9e8-4f46-8349-dba65654eb5d" containerID="d330717b0ea64a64d7b6567b008b2b80a3d3e4705a3ff6fcd5cc27db781b6ecc" exitCode=0
Oct 31 13:09:14.229783 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:14.229765    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p" event={"ID":"98c5973c-b9e8-4f46-8349-dba65654eb5d","Type":"ContainerDied","Data":"d330717b0ea64a64d7b6567b008b2b80a3d3e4705a3ff6fcd5cc27db781b6ecc"}
Oct 31 13:09:14.234419 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:14.233665    4660 generic.go:356] "Generic (PLEG): container finished" podID="b84c453d-748d-4287-805c-09aa3dbcced8" containerID="07ecdb5d145392e49fddf77d3b4f6ff802450ee62d5aa304983e9f318516ed4f" exitCode=0
Oct 31 13:09:14.234419 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:14.233757    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj" event={"ID":"b84c453d-748d-4287-805c-09aa3dbcced8","Type":"ContainerDied","Data":"07ecdb5d145392e49fddf77d3b4f6ff802450ee62d5aa304983e9f318516ed4f"}
Oct 31 13:09:20.815363 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:09:20.815096    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:09:21.356353 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:21.356270    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q" event={"ID":"dbce6e87-58e5-4f07-bdd2-5f2eb402836a","Type":"ContainerStarted","Data":"343e5a659fae1222509eaa386d1c86fbc72a726226ab1f56a85cc0e1369fb25d"}
Oct 31 13:09:22.396604 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:22.395019    4660 generic.go:356] "Generic (PLEG): container finished" podID="dbce6e87-58e5-4f07-bdd2-5f2eb402836a" containerID="343e5a659fae1222509eaa386d1c86fbc72a726226ab1f56a85cc0e1369fb25d" exitCode=0
Oct 31 13:09:22.396604 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:22.395177    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q" event={"ID":"dbce6e87-58e5-4f07-bdd2-5f2eb402836a","Type":"ContainerDied","Data":"343e5a659fae1222509eaa386d1c86fbc72a726226ab1f56a85cc0e1369fb25d"}
Oct 31 13:09:32.522596 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:32.522509    4660 generic.go:356] "Generic (PLEG): container finished" podID="91930a15-be9e-4cd0-ad46-1b367d06e8fa" containerID="44380d12d422eeec509da3b1e3f3b8468be72a6dcf69c1890cad29a3aa542f3c" exitCode=0
Oct 31 13:09:32.529355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:32.529319    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d" event={"ID":"91930a15-be9e-4cd0-ad46-1b367d06e8fa","Type":"ContainerDied","Data":"44380d12d422eeec509da3b1e3f3b8468be72a6dcf69c1890cad29a3aa542f3c"}
Oct 31 13:09:33.543192 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:33.543155    4660 generic.go:356] "Generic (PLEG): container finished" podID="345cc59b-0448-4ecb-ac99-2d9b68c1744b" containerID="d13943f267be73f2c5a150d5398d5132735988bcebcd169c232969d562db0671" exitCode=0
Oct 31 13:09:33.543993 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:33.543918    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4" event={"ID":"345cc59b-0448-4ecb-ac99-2d9b68c1744b","Type":"ContainerDied","Data":"d13943f267be73f2c5a150d5398d5132735988bcebcd169c232969d562db0671"}
Oct 31 13:09:33.815944 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:33.815569    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d"
Oct 31 13:09:33.890099 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:33.889287    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91930a15-be9e-4cd0-ad46-1b367d06e8fa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/91930a15-be9e-4cd0-ad46-1b367d06e8fa-bundle\") pod \"91930a15-be9e-4cd0-ad46-1b367d06e8fa\" (UID: \"91930a15-be9e-4cd0-ad46-1b367d06e8fa\") "
Oct 31 13:09:33.890099 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:33.889381    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91930a15-be9e-4cd0-ad46-1b367d06e8fa-util\" (UniqueName: \"kubernetes.io/empty-dir/91930a15-be9e-4cd0-ad46-1b367d06e8fa-util\") pod \"91930a15-be9e-4cd0-ad46-1b367d06e8fa\" (UID: \"91930a15-be9e-4cd0-ad46-1b367d06e8fa\") "
Oct 31 13:09:33.890099 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:33.889427    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91930a15-be9e-4cd0-ad46-1b367d06e8fa-kube-api-access-fgjhq\" (UniqueName: \"kubernetes.io/projected/91930a15-be9e-4cd0-ad46-1b367d06e8fa-kube-api-access-fgjhq\") pod \"91930a15-be9e-4cd0-ad46-1b367d06e8fa\" (UID: \"91930a15-be9e-4cd0-ad46-1b367d06e8fa\") "
Oct 31 13:09:33.891152 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:33.890805    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91930a15-be9e-4cd0-ad46-1b367d06e8fa-bundle" pod "91930a15-be9e-4cd0-ad46-1b367d06e8fa" (UID: "91930a15-be9e-4cd0-ad46-1b367d06e8fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:33.896618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:33.896573    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91930a15-be9e-4cd0-ad46-1b367d06e8fa-kube-api-access-fgjhq" pod "91930a15-be9e-4cd0-ad46-1b367d06e8fa" (UID: "91930a15-be9e-4cd0-ad46-1b367d06e8fa"). InnerVolumeSpecName "kube-api-access-fgjhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:09:33.896913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:33.896894    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91930a15-be9e-4cd0-ad46-1b367d06e8fa-util" pod "91930a15-be9e-4cd0-ad46-1b367d06e8fa" (UID: "91930a15-be9e-4cd0-ad46-1b367d06e8fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:33.990546 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:33.990194    4660 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91930a15-be9e-4cd0-ad46-1b367d06e8fa-bundle\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:33.990546 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:33.990228    4660 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91930a15-be9e-4cd0-ad46-1b367d06e8fa-util\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:33.990546 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:33.990244    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgjhq\" (UniqueName: \"kubernetes.io/projected/91930a15-be9e-4cd0-ad46-1b367d06e8fa-kube-api-access-fgjhq\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:34.566549 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:34.566520    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d"
Oct 31 13:09:34.567861 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:34.567782    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69w98d" event={"ID":"91930a15-be9e-4cd0-ad46-1b367d06e8fa","Type":"ContainerDied","Data":"29026baa85a7185d0cf2af50dab39ab35da9c286c500be9951a36a775c294551"}
Oct 31 13:09:34.567861 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:34.567832    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29026baa85a7185d0cf2af50dab39ab35da9c286c500be9951a36a775c294551"
Oct 31 13:09:34.786155 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:34.786116    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4"
Oct 31 13:09:34.899349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:34.899289    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/345cc59b-0448-4ecb-ac99-2d9b68c1744b-kube-api-access-6hp6m\" (UniqueName: \"kubernetes.io/projected/345cc59b-0448-4ecb-ac99-2d9b68c1744b-kube-api-access-6hp6m\") pod \"345cc59b-0448-4ecb-ac99-2d9b68c1744b\" (UID: \"345cc59b-0448-4ecb-ac99-2d9b68c1744b\") "
Oct 31 13:09:34.899531 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:34.899361    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345cc59b-0448-4ecb-ac99-2d9b68c1744b-util\" (UniqueName: \"kubernetes.io/empty-dir/345cc59b-0448-4ecb-ac99-2d9b68c1744b-util\") pod \"345cc59b-0448-4ecb-ac99-2d9b68c1744b\" (UID: \"345cc59b-0448-4ecb-ac99-2d9b68c1744b\") "
Oct 31 13:09:34.899531 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:34.899394    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345cc59b-0448-4ecb-ac99-2d9b68c1744b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/345cc59b-0448-4ecb-ac99-2d9b68c1744b-bundle\") pod \"345cc59b-0448-4ecb-ac99-2d9b68c1744b\" (UID: \"345cc59b-0448-4ecb-ac99-2d9b68c1744b\") "
Oct 31 13:09:34.900696 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:34.900639    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345cc59b-0448-4ecb-ac99-2d9b68c1744b-bundle" pod "345cc59b-0448-4ecb-ac99-2d9b68c1744b" (UID: "345cc59b-0448-4ecb-ac99-2d9b68c1744b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:34.904611 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:34.904583    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345cc59b-0448-4ecb-ac99-2d9b68c1744b-util" pod "345cc59b-0448-4ecb-ac99-2d9b68c1744b" (UID: "345cc59b-0448-4ecb-ac99-2d9b68c1744b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:34.910478 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:34.910430    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345cc59b-0448-4ecb-ac99-2d9b68c1744b-kube-api-access-6hp6m" pod "345cc59b-0448-4ecb-ac99-2d9b68c1744b" (UID: "345cc59b-0448-4ecb-ac99-2d9b68c1744b"). InnerVolumeSpecName "kube-api-access-6hp6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:09:35.000923 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:35.000882    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hp6m\" (UniqueName: \"kubernetes.io/projected/345cc59b-0448-4ecb-ac99-2d9b68c1744b-kube-api-access-6hp6m\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:35.000923 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:35.000918    4660 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/345cc59b-0448-4ecb-ac99-2d9b68c1744b-util\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:35.000923 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:35.000931    4660 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/345cc59b-0448-4ecb-ac99-2d9b68c1744b-bundle\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:35.580910 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:35.580864    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4" event={"ID":"345cc59b-0448-4ecb-ac99-2d9b68c1744b","Type":"ContainerDied","Data":"01f41099b7ce749aa48da478b9125531d5ebd22d26f52aa758da7d179782b002"}
Oct 31 13:09:35.581352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:35.580932    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01f41099b7ce749aa48da478b9125531d5ebd22d26f52aa758da7d179782b002"
Oct 31 13:09:35.581352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:35.581013    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2lj4"
Oct 31 13:09:36.605738 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:36.605702    4660 generic.go:356] "Generic (PLEG): container finished" podID="67af19e2-81cf-4d1a-9507-454480e4953a" containerID="71df0a501588f9c6cfae9a0d81628ac2604418ddf21f66dadc4e82506f632d64" exitCode=0
Oct 31 13:09:36.606182 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:36.606157    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c" event={"ID":"67af19e2-81cf-4d1a-9507-454480e4953a","Type":"ContainerDied","Data":"71df0a501588f9c6cfae9a0d81628ac2604418ddf21f66dadc4e82506f632d64"}
Oct 31 13:09:36.610510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:36.608991    4660 generic.go:356] "Generic (PLEG): container finished" podID="dbce6e87-58e5-4f07-bdd2-5f2eb402836a" containerID="e10c3d051b8a547541a67bf57193e323de3741b574d11ed277bae39eace6b7cf" exitCode=0
Oct 31 13:09:36.610510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:36.609104    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q" event={"ID":"dbce6e87-58e5-4f07-bdd2-5f2eb402836a","Type":"ContainerDied","Data":"e10c3d051b8a547541a67bf57193e323de3741b574d11ed277bae39eace6b7cf"}
Oct 31 13:09:36.612796 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:36.612769    4660 generic.go:356] "Generic (PLEG): container finished" podID="986b9082-b05e-4fda-b543-40b5931c2df0" containerID="8294e2daa608ef8491d9332bd9df383328524f73cdf401b8fe6bd5f7a97228b4" exitCode=0
Oct 31 13:09:36.612905 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:36.612848    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf" event={"ID":"986b9082-b05e-4fda-b543-40b5931c2df0","Type":"ContainerDied","Data":"8294e2daa608ef8491d9332bd9df383328524f73cdf401b8fe6bd5f7a97228b4"}
Oct 31 13:09:36.615057 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:36.615037    4660 generic.go:356] "Generic (PLEG): container finished" podID="f8494cb1-cd4f-494e-bb4a-4ad0cb406e79" containerID="58b5909e7380d1eaa294dbd251244ff5c99befd0f32ad1cce98e3cbb8c386b99" exitCode=0
Oct 31 13:09:36.615141 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:36.615123    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl" event={"ID":"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79","Type":"ContainerDied","Data":"58b5909e7380d1eaa294dbd251244ff5c99befd0f32ad1cce98e3cbb8c386b99"}
Oct 31 13:09:37.621843 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.621805    4660 generic.go:356] "Generic (PLEG): container finished" podID="98c5973c-b9e8-4f46-8349-dba65654eb5d" containerID="d38bed23c418e18335fb702e5cd8469ace4403d2a57587ab3bca0970c545c770" exitCode=0
Oct 31 13:09:37.622414 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.621897    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p" event={"ID":"98c5973c-b9e8-4f46-8349-dba65654eb5d","Type":"ContainerDied","Data":"d38bed23c418e18335fb702e5cd8469ace4403d2a57587ab3bca0970c545c770"}
Oct 31 13:09:37.623474 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.623456    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:09:37.624077 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.624061    4660 generic.go:356] "Generic (PLEG): container finished" podID="b84c453d-748d-4287-805c-09aa3dbcced8" containerID="480b3d5da6eb01eec3e28ae2177ba99d1c0247cd30a341244b8fd287afdba3ae" exitCode=1
Oct 31 13:09:37.624167 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.624146    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj" event={"ID":"b84c453d-748d-4287-805c-09aa3dbcced8","Type":"ContainerDied","Data":"480b3d5da6eb01eec3e28ae2177ba99d1c0247cd30a341244b8fd287afdba3ae"}
Oct 31 13:09:37.626090 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.626070    4660 generic.go:356] "Generic (PLEG): container finished" podID="948938f2-e87c-4588-9803-d7886020f8ca" containerID="e417434d66a4c82a8c2135c65b42ca1728680a0113675e7dc8349ac71e782558" exitCode=0
Oct 31 13:09:37.626186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.626102    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj" event={"ID":"948938f2-e87c-4588-9803-d7886020f8ca","Type":"ContainerDied","Data":"e417434d66a4c82a8c2135c65b42ca1728680a0113675e7dc8349ac71e782558"}
Oct 31 13:09:37.628060 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.628042    4660 generic.go:356] "Generic (PLEG): container finished" podID="b5e713c6-941d-4219-afde-75e3d40feb51" containerID="8d301e3b141be0804bfd4b86b5e35280448d14a43cf750848b808e07f9354dd9" exitCode=0
Oct 31 13:09:37.628153 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.628066    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf" event={"ID":"b5e713c6-941d-4219-afde-75e3d40feb51","Type":"ContainerDied","Data":"8d301e3b141be0804bfd4b86b5e35280448d14a43cf750848b808e07f9354dd9"}
Oct 31 13:09:37.629577 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.629562    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:09:37.630198 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.630181    4660 generic.go:356] "Generic (PLEG): container finished" podID="2e19280d-a86c-48cd-b4e1-7cbe8308cb7d" containerID="766bf09360af4fc3208d46a963b7d6b2665d1f63f8ea0ea601cb231ef25f773d" exitCode=1
Oct 31 13:09:37.630311 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.630272    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49" event={"ID":"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d","Type":"ContainerDied","Data":"766bf09360af4fc3208d46a963b7d6b2665d1f63f8ea0ea601cb231ef25f773d"}
Oct 31 13:09:37.778476 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.778452    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q"
Oct 31 13:09:37.818511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.818483    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl"
Oct 31 13:09:37.821444 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.821422    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-bundle\") pod \"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79\" (UID: \"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79\") "
Oct 31 13:09:37.821526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.821460    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-util\" (UniqueName: \"kubernetes.io/empty-dir/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-util\") pod \"dbce6e87-58e5-4f07-bdd2-5f2eb402836a\" (UID: \"dbce6e87-58e5-4f07-bdd2-5f2eb402836a\") "
Oct 31 13:09:37.821526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.821510    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-kube-api-access-tzq9r\" (UniqueName: \"kubernetes.io/projected/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-kube-api-access-tzq9r\") pod \"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79\" (UID: \"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79\") "
Oct 31 13:09:37.821609 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.821555    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-util\" (UniqueName: \"kubernetes.io/empty-dir/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-util\") pod \"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79\" (UID: \"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79\") "
Oct 31 13:09:37.821609 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.821586    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-bundle\") pod \"dbce6e87-58e5-4f07-bdd2-5f2eb402836a\" (UID: \"dbce6e87-58e5-4f07-bdd2-5f2eb402836a\") "
Oct 31 13:09:37.821720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.821682    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-kube-api-access-dtpgp\" (UniqueName: \"kubernetes.io/projected/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-kube-api-access-dtpgp\") pod \"dbce6e87-58e5-4f07-bdd2-5f2eb402836a\" (UID: \"dbce6e87-58e5-4f07-bdd2-5f2eb402836a\") "
Oct 31 13:09:37.821858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.821831    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-bundle" pod "f8494cb1-cd4f-494e-bb4a-4ad0cb406e79" (UID: "f8494cb1-cd4f-494e-bb4a-4ad0cb406e79"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:37.821944 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.821924    4660 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-bundle\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:37.822712 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.822686    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-bundle" pod "dbce6e87-58e5-4f07-bdd2-5f2eb402836a" (UID: "dbce6e87-58e5-4f07-bdd2-5f2eb402836a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:37.826828 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.826800    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-kube-api-access-tzq9r" pod "f8494cb1-cd4f-494e-bb4a-4ad0cb406e79" (UID: "f8494cb1-cd4f-494e-bb4a-4ad0cb406e79"). InnerVolumeSpecName "kube-api-access-tzq9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:09:37.828041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.828010    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-kube-api-access-dtpgp" pod "dbce6e87-58e5-4f07-bdd2-5f2eb402836a" (UID: "dbce6e87-58e5-4f07-bdd2-5f2eb402836a"). InnerVolumeSpecName "kube-api-access-dtpgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:09:37.829467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.829438    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-util" pod "f8494cb1-cd4f-494e-bb4a-4ad0cb406e79" (UID: "f8494cb1-cd4f-494e-bb4a-4ad0cb406e79"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:37.830161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.830137    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-util" pod "dbce6e87-58e5-4f07-bdd2-5f2eb402836a" (UID: "dbce6e87-58e5-4f07-bdd2-5f2eb402836a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:37.851578 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.851545    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c"
Oct 31 13:09:37.859338 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.859312    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf"
Oct 31 13:09:37.922648 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.922572    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67af19e2-81cf-4d1a-9507-454480e4953a-util\" (UniqueName: \"kubernetes.io/empty-dir/67af19e2-81cf-4d1a-9507-454480e4953a-util\") pod \"67af19e2-81cf-4d1a-9507-454480e4953a\" (UID: \"67af19e2-81cf-4d1a-9507-454480e4953a\") "
Oct 31 13:09:37.922648 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.922620    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67af19e2-81cf-4d1a-9507-454480e4953a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/67af19e2-81cf-4d1a-9507-454480e4953a-bundle\") pod \"67af19e2-81cf-4d1a-9507-454480e4953a\" (UID: \"67af19e2-81cf-4d1a-9507-454480e4953a\") "
Oct 31 13:09:37.922648 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.922640    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/986b9082-b05e-4fda-b543-40b5931c2df0-kube-api-access-vbvwg\" (UniqueName: \"kubernetes.io/projected/986b9082-b05e-4fda-b543-40b5931c2df0-kube-api-access-vbvwg\") pod \"986b9082-b05e-4fda-b543-40b5931c2df0\" (UID: \"986b9082-b05e-4fda-b543-40b5931c2df0\") "
Oct 31 13:09:37.922903 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.922662    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986b9082-b05e-4fda-b543-40b5931c2df0-util\" (UniqueName: \"kubernetes.io/empty-dir/986b9082-b05e-4fda-b543-40b5931c2df0-util\") pod \"986b9082-b05e-4fda-b543-40b5931c2df0\" (UID: \"986b9082-b05e-4fda-b543-40b5931c2df0\") "
Oct 31 13:09:37.922903 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.922686    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986b9082-b05e-4fda-b543-40b5931c2df0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/986b9082-b05e-4fda-b543-40b5931c2df0-bundle\") pod \"986b9082-b05e-4fda-b543-40b5931c2df0\" (UID: \"986b9082-b05e-4fda-b543-40b5931c2df0\") "
Oct 31 13:09:37.922903 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.922730    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67af19e2-81cf-4d1a-9507-454480e4953a-kube-api-access-hqh68\" (UniqueName: \"kubernetes.io/projected/67af19e2-81cf-4d1a-9507-454480e4953a-kube-api-access-hqh68\") pod \"67af19e2-81cf-4d1a-9507-454480e4953a\" (UID: \"67af19e2-81cf-4d1a-9507-454480e4953a\") "
Oct 31 13:09:37.923056 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.923016    4660 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-util\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:37.923056 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.923045    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzq9r\" (UniqueName: \"kubernetes.io/projected/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-kube-api-access-tzq9r\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:37.923155 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.923059    4660 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8494cb1-cd4f-494e-bb4a-4ad0cb406e79-util\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:37.923155 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.923073    4660 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-bundle\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:37.923155 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.923089    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtpgp\" (UniqueName: \"kubernetes.io/projected/dbce6e87-58e5-4f07-bdd2-5f2eb402836a-kube-api-access-dtpgp\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:37.923404 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.923378    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986b9082-b05e-4fda-b543-40b5931c2df0-bundle" pod "986b9082-b05e-4fda-b543-40b5931c2df0" (UID: "986b9082-b05e-4fda-b543-40b5931c2df0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:37.923465 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.923396    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67af19e2-81cf-4d1a-9507-454480e4953a-bundle" pod "67af19e2-81cf-4d1a-9507-454480e4953a" (UID: "67af19e2-81cf-4d1a-9507-454480e4953a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:37.924931 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.924905    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67af19e2-81cf-4d1a-9507-454480e4953a-kube-api-access-hqh68" pod "67af19e2-81cf-4d1a-9507-454480e4953a" (UID: "67af19e2-81cf-4d1a-9507-454480e4953a"). InnerVolumeSpecName "kube-api-access-hqh68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:09:37.924984 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.924951    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986b9082-b05e-4fda-b543-40b5931c2df0-kube-api-access-vbvwg" pod "986b9082-b05e-4fda-b543-40b5931c2df0" (UID: "986b9082-b05e-4fda-b543-40b5931c2df0"). InnerVolumeSpecName "kube-api-access-vbvwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:09:37.931507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.931477    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986b9082-b05e-4fda-b543-40b5931c2df0-util" pod "986b9082-b05e-4fda-b543-40b5931c2df0" (UID: "986b9082-b05e-4fda-b543-40b5931c2df0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:37.931739 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:37.931720    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67af19e2-81cf-4d1a-9507-454480e4953a-util" pod "67af19e2-81cf-4d1a-9507-454480e4953a" (UID: "67af19e2-81cf-4d1a-9507-454480e4953a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:38.023514 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.023478    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqh68\" (UniqueName: \"kubernetes.io/projected/67af19e2-81cf-4d1a-9507-454480e4953a-kube-api-access-hqh68\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:38.023514 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.023506    4660 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67af19e2-81cf-4d1a-9507-454480e4953a-util\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:38.023514 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.023516    4660 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67af19e2-81cf-4d1a-9507-454480e4953a-bundle\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:38.023733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.023524    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbvwg\" (UniqueName: \"kubernetes.io/projected/986b9082-b05e-4fda-b543-40b5931c2df0-kube-api-access-vbvwg\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:38.023733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.023535    4660 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/986b9082-b05e-4fda-b543-40b5931c2df0-util\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:38.023733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.023544    4660 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/986b9082-b05e-4fda-b543-40b5931c2df0-bundle\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:38.636983 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.636952    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf"
Oct 31 13:09:38.636983 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.636968    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cftdqf" event={"ID":"986b9082-b05e-4fda-b543-40b5931c2df0","Type":"ContainerDied","Data":"f2b9dd5ce118561e6283b0d3104f56c86008d0288d05435e4cd34ff0de6885ee"}
Oct 31 13:09:38.637458 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.637009    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2b9dd5ce118561e6283b0d3104f56c86008d0288d05435e4cd34ff0de6885ee"
Oct 31 13:09:38.639177 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.639102    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl"
Oct 31 13:09:38.639177 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.639133    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448tjcl" event={"ID":"f8494cb1-cd4f-494e-bb4a-4ad0cb406e79","Type":"ContainerDied","Data":"f1b71843b3585ee1fdd7e1f1285428670b147658f7cb12da0b553798b69f2a7a"}
Oct 31 13:09:38.639177 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.639171    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1b71843b3585ee1fdd7e1f1285428670b147658f7cb12da0b553798b69f2a7a"
Oct 31 13:09:38.641470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.641450    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c"
Oct 31 13:09:38.641568 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.641448    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715qj7c" event={"ID":"67af19e2-81cf-4d1a-9507-454480e4953a","Type":"ContainerDied","Data":"562cc68c7d2bc63e48d7eeda3730b64f9074feaeffda2c444de745c7b2c8258b"}
Oct 31 13:09:38.641568 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.641535    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="562cc68c7d2bc63e48d7eeda3730b64f9074feaeffda2c444de745c7b2c8258b"
Oct 31 13:09:38.643501 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.643475    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q" event={"ID":"dbce6e87-58e5-4f07-bdd2-5f2eb402836a","Type":"ContainerDied","Data":"dda6f6cf76da37c557250722715c8e4b75f5dd570ee7ab48ee7465de00954612"}
Oct 31 13:09:38.643598 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.643528    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dda6f6cf76da37c557250722715c8e4b75f5dd570ee7ab48ee7465de00954612"
Oct 31 13:09:38.643749 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.643731    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crqz7q"
Oct 31 13:09:38.792760 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.792691    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:09:38.799723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.799696    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj"
Oct 31 13:09:38.872178 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.872157    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:09:38.875382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.875361    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49"
Oct 31 13:09:38.879581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.879562    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf"
Oct 31 13:09:38.883254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.883234    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p"
Oct 31 13:09:38.886145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.886129    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj"
Oct 31 13:09:38.931305 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.931265    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948938f2-e87c-4588-9803-d7886020f8ca-bundle\" (UniqueName: \"kubernetes.io/empty-dir/948938f2-e87c-4588-9803-d7886020f8ca-bundle\") pod \"948938f2-e87c-4588-9803-d7886020f8ca\" (UID: \"948938f2-e87c-4588-9803-d7886020f8ca\") "
Oct 31 13:09:38.931453 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.931335    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5e713c6-941d-4219-afde-75e3d40feb51-util\" (UniqueName: \"kubernetes.io/empty-dir/b5e713c6-941d-4219-afde-75e3d40feb51-util\") pod \"b5e713c6-941d-4219-afde-75e3d40feb51\" (UID: \"b5e713c6-941d-4219-afde-75e3d40feb51\") "
Oct 31 13:09:38.931453 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.931384    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b84c453d-748d-4287-805c-09aa3dbcced8-kube-api-access-xx6tg\" (UniqueName: \"kubernetes.io/projected/b84c453d-748d-4287-805c-09aa3dbcced8-kube-api-access-xx6tg\") pod \"b84c453d-748d-4287-805c-09aa3dbcced8\" (UID: \"b84c453d-748d-4287-805c-09aa3dbcced8\") "
Oct 31 13:09:38.931453 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.931424    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84c453d-748d-4287-805c-09aa3dbcced8-util\" (UniqueName: \"kubernetes.io/empty-dir/b84c453d-748d-4287-805c-09aa3dbcced8-util\") pod \"b84c453d-748d-4287-805c-09aa3dbcced8\" (UID: \"b84c453d-748d-4287-805c-09aa3dbcced8\") "
Oct 31 13:09:38.931616 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.931452    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-util\" (UniqueName: \"kubernetes.io/empty-dir/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-util\") pod \"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d\" (UID: \"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d\") "
Oct 31 13:09:38.931616 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.931480    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-bundle\") pod \"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d\" (UID: \"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d\") "
Oct 31 13:09:38.931616 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.931505    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c5973c-b9e8-4f46-8349-dba65654eb5d-util\" (UniqueName: \"kubernetes.io/empty-dir/98c5973c-b9e8-4f46-8349-dba65654eb5d-util\") pod \"98c5973c-b9e8-4f46-8349-dba65654eb5d\" (UID: \"98c5973c-b9e8-4f46-8349-dba65654eb5d\") "
Oct 31 13:09:38.931616 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.931534    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948938f2-e87c-4588-9803-d7886020f8ca-kube-api-access-t2jng\" (UniqueName: \"kubernetes.io/projected/948938f2-e87c-4588-9803-d7886020f8ca-kube-api-access-t2jng\") pod \"948938f2-e87c-4588-9803-d7886020f8ca\" (UID: \"948938f2-e87c-4588-9803-d7886020f8ca\") "
Oct 31 13:09:38.931616 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.931582    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c5973c-b9e8-4f46-8349-dba65654eb5d-kube-api-access-fl7dc\" (UniqueName: \"kubernetes.io/projected/98c5973c-b9e8-4f46-8349-dba65654eb5d-kube-api-access-fl7dc\") pod \"98c5973c-b9e8-4f46-8349-dba65654eb5d\" (UID: \"98c5973c-b9e8-4f46-8349-dba65654eb5d\") "
Oct 31 13:09:38.931855 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.931622    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84c453d-748d-4287-805c-09aa3dbcced8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b84c453d-748d-4287-805c-09aa3dbcced8-bundle\") pod \"b84c453d-748d-4287-805c-09aa3dbcced8\" (UID: \"b84c453d-748d-4287-805c-09aa3dbcced8\") "
Oct 31 13:09:38.931855 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.931651    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e713c6-941d-4219-afde-75e3d40feb51-kube-api-access-n9jqm\" (UniqueName: \"kubernetes.io/projected/b5e713c6-941d-4219-afde-75e3d40feb51-kube-api-access-n9jqm\") pod \"b5e713c6-941d-4219-afde-75e3d40feb51\" (UID: \"b5e713c6-941d-4219-afde-75e3d40feb51\") "
Oct 31 13:09:38.931989 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.931956    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948938f2-e87c-4588-9803-d7886020f8ca-bundle" pod "948938f2-e87c-4588-9803-d7886020f8ca" (UID: "948938f2-e87c-4588-9803-d7886020f8ca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:38.932832 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.932582    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-bundle" pod "2e19280d-a86c-48cd-b4e1-7cbe8308cb7d" (UID: "2e19280d-a86c-48cd-b4e1-7cbe8308cb7d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:38.934655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.934285    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948938f2-e87c-4588-9803-d7886020f8ca-kube-api-access-t2jng" pod "948938f2-e87c-4588-9803-d7886020f8ca" (UID: "948938f2-e87c-4588-9803-d7886020f8ca"). InnerVolumeSpecName "kube-api-access-t2jng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:09:38.934964 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.934940    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84c453d-748d-4287-805c-09aa3dbcced8-bundle" pod "b84c453d-748d-4287-805c-09aa3dbcced8" (UID: "b84c453d-748d-4287-805c-09aa3dbcced8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:38.935474 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.935452    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e713c6-941d-4219-afde-75e3d40feb51-kube-api-access-n9jqm" pod "b5e713c6-941d-4219-afde-75e3d40feb51" (UID: "b5e713c6-941d-4219-afde-75e3d40feb51"). InnerVolumeSpecName "kube-api-access-n9jqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:09:38.935659 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.935624    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c5973c-b9e8-4f46-8349-dba65654eb5d-kube-api-access-fl7dc" pod "98c5973c-b9e8-4f46-8349-dba65654eb5d" (UID: "98c5973c-b9e8-4f46-8349-dba65654eb5d"). InnerVolumeSpecName "kube-api-access-fl7dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:09:38.935786 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.935762    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84c453d-748d-4287-805c-09aa3dbcced8-kube-api-access-xx6tg" pod "b84c453d-748d-4287-805c-09aa3dbcced8" (UID: "b84c453d-748d-4287-805c-09aa3dbcced8"). InnerVolumeSpecName "kube-api-access-xx6tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:09:38.940225 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.940193    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-util" pod "2e19280d-a86c-48cd-b4e1-7cbe8308cb7d" (UID: "2e19280d-a86c-48cd-b4e1-7cbe8308cb7d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:38.940471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.940440    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84c453d-748d-4287-805c-09aa3dbcced8-util" pod "b84c453d-748d-4287-805c-09aa3dbcced8" (UID: "b84c453d-748d-4287-805c-09aa3dbcced8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:38.941227 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.941200    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c5973c-b9e8-4f46-8349-dba65654eb5d-util" pod "98c5973c-b9e8-4f46-8349-dba65654eb5d" (UID: "98c5973c-b9e8-4f46-8349-dba65654eb5d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:38.941871 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:38.941842    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5e713c6-941d-4219-afde-75e3d40feb51-util" pod "b5e713c6-941d-4219-afde-75e3d40feb51" (UID: "b5e713c6-941d-4219-afde-75e3d40feb51"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:39.032914 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.032874    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-kube-api-access-k9pmf\" (UniqueName: \"kubernetes.io/projected/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-kube-api-access-k9pmf\") pod \"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d\" (UID: \"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d\") "
Oct 31 13:09:39.032914 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.032916    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5e713c6-941d-4219-afde-75e3d40feb51-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5e713c6-941d-4219-afde-75e3d40feb51-bundle\") pod \"b5e713c6-941d-4219-afde-75e3d40feb51\" (UID: \"b5e713c6-941d-4219-afde-75e3d40feb51\") "
Oct 31 13:09:39.033186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.032938    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948938f2-e87c-4588-9803-d7886020f8ca-util\" (UniqueName: \"kubernetes.io/empty-dir/948938f2-e87c-4588-9803-d7886020f8ca-util\") pod \"948938f2-e87c-4588-9803-d7886020f8ca\" (UID: \"948938f2-e87c-4588-9803-d7886020f8ca\") "
Oct 31 13:09:39.033186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.032959    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c5973c-b9e8-4f46-8349-dba65654eb5d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/98c5973c-b9e8-4f46-8349-dba65654eb5d-bundle\") pod \"98c5973c-b9e8-4f46-8349-dba65654eb5d\" (UID: \"98c5973c-b9e8-4f46-8349-dba65654eb5d\") "
Oct 31 13:09:39.033186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.033161    4660 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/948938f2-e87c-4588-9803-d7886020f8ca-bundle\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:39.033186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.033173    4660 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5e713c6-941d-4219-afde-75e3d40feb51-util\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:39.033186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.033183    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx6tg\" (UniqueName: \"kubernetes.io/projected/b84c453d-748d-4287-805c-09aa3dbcced8-kube-api-access-xx6tg\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:39.033467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.033193    4660 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b84c453d-748d-4287-805c-09aa3dbcced8-util\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:39.033467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.033204    4660 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-util\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:39.033467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.033211    4660 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-bundle\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:39.033467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.033219    4660 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98c5973c-b9e8-4f46-8349-dba65654eb5d-util\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:39.033467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.033227    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2jng\" (UniqueName: \"kubernetes.io/projected/948938f2-e87c-4588-9803-d7886020f8ca-kube-api-access-t2jng\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:39.033467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.033235    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl7dc\" (UniqueName: \"kubernetes.io/projected/98c5973c-b9e8-4f46-8349-dba65654eb5d-kube-api-access-fl7dc\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:39.033467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.033243    4660 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b84c453d-748d-4287-805c-09aa3dbcced8-bundle\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:39.033467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.033251    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9jqm\" (UniqueName: \"kubernetes.io/projected/b5e713c6-941d-4219-afde-75e3d40feb51-kube-api-access-n9jqm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:39.033820 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.033539    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c5973c-b9e8-4f46-8349-dba65654eb5d-bundle" pod "98c5973c-b9e8-4f46-8349-dba65654eb5d" (UID: "98c5973c-b9e8-4f46-8349-dba65654eb5d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:39.033820 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.033700    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5e713c6-941d-4219-afde-75e3d40feb51-bundle" pod "b5e713c6-941d-4219-afde-75e3d40feb51" (UID: "b5e713c6-941d-4219-afde-75e3d40feb51"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:39.035041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.035016    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-kube-api-access-k9pmf" pod "2e19280d-a86c-48cd-b4e1-7cbe8308cb7d" (UID: "2e19280d-a86c-48cd-b4e1-7cbe8308cb7d"). InnerVolumeSpecName "kube-api-access-k9pmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:09:39.038834 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.038808    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948938f2-e87c-4588-9803-d7886020f8ca-util" pod "948938f2-e87c-4588-9803-d7886020f8ca" (UID: "948938f2-e87c-4588-9803-d7886020f8ca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:39.133758 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.133725    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9pmf\" (UniqueName: \"kubernetes.io/projected/2e19280d-a86c-48cd-b4e1-7cbe8308cb7d-kube-api-access-k9pmf\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:39.133758 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.133758    4660 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5e713c6-941d-4219-afde-75e3d40feb51-bundle\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:39.133943 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.133772    4660 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/948938f2-e87c-4588-9803-d7886020f8ca-util\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:39.133943 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.133783    4660 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98c5973c-b9e8-4f46-8349-dba65654eb5d-bundle\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:39.650390 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.650364    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj"
Oct 31 13:09:39.650807 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.650360    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czv9nj" event={"ID":"948938f2-e87c-4588-9803-d7886020f8ca","Type":"ContainerDied","Data":"81a4bcb2d290e03d3e369a78530531284f18e00c98b73e0c8b3358be98aa8d68"}
Oct 31 13:09:39.650807 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.650488    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81a4bcb2d290e03d3e369a78530531284f18e00c98b73e0c8b3358be98aa8d68"
Oct 31 13:09:39.652435 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.652404    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf"
Oct 31 13:09:39.652561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.652403    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afnbgf" event={"ID":"b5e713c6-941d-4219-afde-75e3d40feb51","Type":"ContainerDied","Data":"3d4477fc347c7f22acdd776eeac17da6024e0ea22570acc9be97576a23c382e4"}
Oct 31 13:09:39.652561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.652520    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d4477fc347c7f22acdd776eeac17da6024e0ea22570acc9be97576a23c382e4"
Oct 31 13:09:39.653789 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.653772    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:09:39.654481 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.654465    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49"
Oct 31 13:09:39.654575 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.654492    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49" event={"ID":"2e19280d-a86c-48cd-b4e1-7cbe8308cb7d","Type":"ContainerDied","Data":"3089bccc3f9daf51556723400d6876c60649e2728b2ae17319fa4be663a78731"}
Oct 31 13:09:39.654575 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.654533    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3089bccc3f9daf51556723400d6876c60649e2728b2ae17319fa4be663a78731"
Oct 31 13:09:39.656650 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.656626    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p" event={"ID":"98c5973c-b9e8-4f46-8349-dba65654eb5d","Type":"ContainerDied","Data":"db2fe3a1413290e0b5550133c120978a2650a1b011b72a0b4f4b975c16474f18"}
Oct 31 13:09:39.656754 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.656652    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hwq2p"
Oct 31 13:09:39.656754 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.656667    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db2fe3a1413290e0b5550133c120978a2650a1b011b72a0b4f4b975c16474f18"
Oct 31 13:09:39.658133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.658115    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:09:39.658860 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.658843    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj"
Oct 31 13:09:39.658944 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.658841    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj" event={"ID":"b84c453d-748d-4287-805c-09aa3dbcced8","Type":"ContainerDied","Data":"e9c57b3aed3c8ef0e45f0bf4685e9a10f4f0f7051c8ad97276d11a95de8d53b6"}
Oct 31 13:09:39.659001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:39.658965    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9c57b3aed3c8ef0e45f0bf4685e9a10f4f0f7051c8ad97276d11a95de8d53b6"
Oct 31 13:09:47.554926 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.554889    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"]
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555193    4660 cpu_manager.go:414] "containerMap: removing container" podUID="2e19280d-a86c-48cd-b4e1-7cbe8308cb7d" containerName="util"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555204    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e19280d-a86c-48cd-b4e1-7cbe8308cb7d" containerName="util"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555214    4660 cpu_manager.go:414] "containerMap: removing container" podUID="986b9082-b05e-4fda-b543-40b5931c2df0" containerName="util"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555219    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="986b9082-b05e-4fda-b543-40b5931c2df0" containerName="util"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555226    4660 cpu_manager.go:414] "containerMap: removing container" podUID="986b9082-b05e-4fda-b543-40b5931c2df0" containerName="pull"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555231    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="986b9082-b05e-4fda-b543-40b5931c2df0" containerName="pull"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555238    4660 cpu_manager.go:414] "containerMap: removing container" podUID="345cc59b-0448-4ecb-ac99-2d9b68c1744b" containerName="util"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555243    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="345cc59b-0448-4ecb-ac99-2d9b68c1744b" containerName="util"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555250    4660 cpu_manager.go:414] "containerMap: removing container" podUID="67af19e2-81cf-4d1a-9507-454480e4953a" containerName="extract"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555254    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67af19e2-81cf-4d1a-9507-454480e4953a" containerName="extract"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555263    4660 cpu_manager.go:414] "containerMap: removing container" podUID="91930a15-be9e-4cd0-ad46-1b367d06e8fa" containerName="util"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555267    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91930a15-be9e-4cd0-ad46-1b367d06e8fa" containerName="util"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555274    4660 cpu_manager.go:414] "containerMap: removing container" podUID="91930a15-be9e-4cd0-ad46-1b367d06e8fa" containerName="pull"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555278    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91930a15-be9e-4cd0-ad46-1b367d06e8fa" containerName="pull"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555283    4660 cpu_manager.go:414] "containerMap: removing container" podUID="2e19280d-a86c-48cd-b4e1-7cbe8308cb7d" containerName="extract"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555287    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e19280d-a86c-48cd-b4e1-7cbe8308cb7d" containerName="extract"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555309    4660 cpu_manager.go:414] "containerMap: removing container" podUID="948938f2-e87c-4588-9803-d7886020f8ca" containerName="extract"
Oct 31 13:09:47.555285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555314    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="948938f2-e87c-4588-9803-d7886020f8ca" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555326    4660 cpu_manager.go:414] "containerMap: removing container" podUID="345cc59b-0448-4ecb-ac99-2d9b68c1744b" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555330    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="345cc59b-0448-4ecb-ac99-2d9b68c1744b" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555337    4660 cpu_manager.go:414] "containerMap: removing container" podUID="67af19e2-81cf-4d1a-9507-454480e4953a" containerName="util"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555343    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67af19e2-81cf-4d1a-9507-454480e4953a" containerName="util"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555350    4660 cpu_manager.go:414] "containerMap: removing container" podUID="dbce6e87-58e5-4f07-bdd2-5f2eb402836a" containerName="util"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555354    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbce6e87-58e5-4f07-bdd2-5f2eb402836a" containerName="util"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555359    4660 cpu_manager.go:414] "containerMap: removing container" podUID="948938f2-e87c-4588-9803-d7886020f8ca" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555363    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="948938f2-e87c-4588-9803-d7886020f8ca" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555368    4660 cpu_manager.go:414] "containerMap: removing container" podUID="dbce6e87-58e5-4f07-bdd2-5f2eb402836a" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555372    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbce6e87-58e5-4f07-bdd2-5f2eb402836a" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555379    4660 cpu_manager.go:414] "containerMap: removing container" podUID="948938f2-e87c-4588-9803-d7886020f8ca" containerName="util"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555383    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="948938f2-e87c-4588-9803-d7886020f8ca" containerName="util"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555388    4660 cpu_manager.go:414] "containerMap: removing container" podUID="98c5973c-b9e8-4f46-8349-dba65654eb5d" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555393    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c5973c-b9e8-4f46-8349-dba65654eb5d" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555400    4660 cpu_manager.go:414] "containerMap: removing container" podUID="f8494cb1-cd4f-494e-bb4a-4ad0cb406e79" containerName="util"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555404    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8494cb1-cd4f-494e-bb4a-4ad0cb406e79" containerName="util"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555410    4660 cpu_manager.go:414] "containerMap: removing container" podUID="b84c453d-748d-4287-805c-09aa3dbcced8" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555414    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b84c453d-748d-4287-805c-09aa3dbcced8" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555421    4660 cpu_manager.go:414] "containerMap: removing container" podUID="986b9082-b05e-4fda-b543-40b5931c2df0" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555425    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="986b9082-b05e-4fda-b543-40b5931c2df0" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555433    4660 cpu_manager.go:414] "containerMap: removing container" podUID="b84c453d-748d-4287-805c-09aa3dbcced8" containerName="util"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555436    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b84c453d-748d-4287-805c-09aa3dbcced8" containerName="util"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555442    4660 cpu_manager.go:414] "containerMap: removing container" podUID="67af19e2-81cf-4d1a-9507-454480e4953a" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555445    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67af19e2-81cf-4d1a-9507-454480e4953a" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555451    4660 cpu_manager.go:414] "containerMap: removing container" podUID="91930a15-be9e-4cd0-ad46-1b367d06e8fa" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555455    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91930a15-be9e-4cd0-ad46-1b367d06e8fa" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555463    4660 cpu_manager.go:414] "containerMap: removing container" podUID="b84c453d-748d-4287-805c-09aa3dbcced8" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555467    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b84c453d-748d-4287-805c-09aa3dbcced8" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555473    4660 cpu_manager.go:414] "containerMap: removing container" podUID="b5e713c6-941d-4219-afde-75e3d40feb51" containerName="util"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555477    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e713c6-941d-4219-afde-75e3d40feb51" containerName="util"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555485    4660 cpu_manager.go:414] "containerMap: removing container" podUID="98c5973c-b9e8-4f46-8349-dba65654eb5d" containerName="util"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555490    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c5973c-b9e8-4f46-8349-dba65654eb5d" containerName="util"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555495    4660 cpu_manager.go:414] "containerMap: removing container" podUID="b5e713c6-941d-4219-afde-75e3d40feb51" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555499    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e713c6-941d-4219-afde-75e3d40feb51" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555505    4660 cpu_manager.go:414] "containerMap: removing container" podUID="f8494cb1-cd4f-494e-bb4a-4ad0cb406e79" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555509    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8494cb1-cd4f-494e-bb4a-4ad0cb406e79" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555515    4660 cpu_manager.go:414] "containerMap: removing container" podUID="b5e713c6-941d-4219-afde-75e3d40feb51" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555520    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e713c6-941d-4219-afde-75e3d40feb51" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555527    4660 cpu_manager.go:414] "containerMap: removing container" podUID="2e19280d-a86c-48cd-b4e1-7cbe8308cb7d" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555531    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e19280d-a86c-48cd-b4e1-7cbe8308cb7d" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555536    4660 cpu_manager.go:414] "containerMap: removing container" podUID="98c5973c-b9e8-4f46-8349-dba65654eb5d" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555540    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c5973c-b9e8-4f46-8349-dba65654eb5d" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555545    4660 cpu_manager.go:414] "containerMap: removing container" podUID="dbce6e87-58e5-4f07-bdd2-5f2eb402836a" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555549    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbce6e87-58e5-4f07-bdd2-5f2eb402836a" containerName="pull"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555554    4660 cpu_manager.go:414] "containerMap: removing container" podUID="f8494cb1-cd4f-494e-bb4a-4ad0cb406e79" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555558    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8494cb1-cd4f-494e-bb4a-4ad0cb406e79" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555565    4660 cpu_manager.go:414] "containerMap: removing container" podUID="345cc59b-0448-4ecb-ac99-2d9b68c1744b" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555569    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="345cc59b-0448-4ecb-ac99-2d9b68c1744b" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555630    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbce6e87-58e5-4f07-bdd2-5f2eb402836a" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555638    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c5973c-b9e8-4f46-8349-dba65654eb5d" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555645    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="986b9082-b05e-4fda-b543-40b5931c2df0" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555651    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="91930a15-be9e-4cd0-ad46-1b367d06e8fa" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555656    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="b84c453d-748d-4287-805c-09aa3dbcced8" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555662    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e713c6-941d-4219-afde-75e3d40feb51" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555667    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8494cb1-cd4f-494e-bb4a-4ad0cb406e79" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555672    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="345cc59b-0448-4ecb-ac99-2d9b68c1744b" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555677    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="67af19e2-81cf-4d1a-9507-454480e4953a" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555682    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e19280d-a86c-48cd-b4e1-7cbe8308cb7d" containerName="extract"
Oct 31 13:09:47.555812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.555688    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="948938f2-e87c-4588-9803-d7886020f8ca" containerName="extract"
Oct 31 13:09:47.558451 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.558435    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.563878 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.563854    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 13:09:47.564003 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.563910    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 13:09:47.564109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.564087    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 13:09:47.565288 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.565272    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 13:09:47.565434 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.565408    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 13:09:47.565540 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.565452    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jj88f\""
Oct 31 13:09:47.590276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.590247    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"]
Oct 31 13:09:47.602011 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.601977    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aadd0aca-43fa-4851-8e0a-c0781bf55f32-manifests\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.602011 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.602013    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aadd0aca-43fa-4851-8e0a-c0781bf55f32-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.602269 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.602040    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aadd0aca-43fa-4851-8e0a-c0781bf55f32-metrics-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.602269 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.602111    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aadd0aca-43fa-4851-8e0a-c0781bf55f32-tmp\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.602269 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.602225    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aadd0aca-43fa-4851-8e0a-c0781bf55f32-webhook-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.602269 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.602245    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m767v\" (UniqueName: \"kubernetes.io/projected/aadd0aca-43fa-4851-8e0a-c0781bf55f32-kube-api-access-m767v\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.702866 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.702826    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aadd0aca-43fa-4851-8e0a-c0781bf55f32-webhook-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.702866 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.702867    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m767v\" (UniqueName: \"kubernetes.io/projected/aadd0aca-43fa-4851-8e0a-c0781bf55f32-kube-api-access-m767v\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.703134 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.702949    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aadd0aca-43fa-4851-8e0a-c0781bf55f32-manifests\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.703134 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.702976    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aadd0aca-43fa-4851-8e0a-c0781bf55f32-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.703134 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.703010    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aadd0aca-43fa-4851-8e0a-c0781bf55f32-metrics-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.703134 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.703029    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aadd0aca-43fa-4851-8e0a-c0781bf55f32-tmp\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.703485 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.703452    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aadd0aca-43fa-4851-8e0a-c0781bf55f32-manifests\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.705433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.705408    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aadd0aca-43fa-4851-8e0a-c0781bf55f32-tmp\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.705565 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.705543    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aadd0aca-43fa-4851-8e0a-c0781bf55f32-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.705685 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.705655    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aadd0aca-43fa-4851-8e0a-c0781bf55f32-metrics-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.705732 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.705693    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aadd0aca-43fa-4851-8e0a-c0781bf55f32-webhook-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.721932 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.721905    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m767v\" (UniqueName: \"kubernetes.io/projected/aadd0aca-43fa-4851-8e0a-c0781bf55f32-kube-api-access-m767v\") pod \"opendatahub-operator-controller-manager-7748d49f5-ldbbp\" (UID: \"aadd0aca-43fa-4851-8e0a-c0781bf55f32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:47.870538 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:47.870508    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:48.170458 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:48.170431    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"]
Oct 31 13:09:48.172783 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:09:48.172757    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadd0aca_43fa_4851_8e0a_c0781bf55f32.slice/crio-5d46b21230ee22ba2ef569bdad042d48af0742e4af44b35896fce26aedcb73ba WatchSource:0}: Error finding container 5d46b21230ee22ba2ef569bdad042d48af0742e4af44b35896fce26aedcb73ba: Status 404 returned error can't find the container with id 5d46b21230ee22ba2ef569bdad042d48af0742e4af44b35896fce26aedcb73ba
Oct 31 13:09:48.701736 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:48.701698    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp" event={"ID":"aadd0aca-43fa-4851-8e0a-c0781bf55f32","Type":"ContainerStarted","Data":"5d46b21230ee22ba2ef569bdad042d48af0742e4af44b35896fce26aedcb73ba"}
Oct 31 13:09:50.734363 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:09:50.734318    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:09:51.717951 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:51.717920    4660 generic.go:356] "Generic (PLEG): container finished" podID="aadd0aca-43fa-4851-8e0a-c0781bf55f32" containerID="429f6b87443d71f9aa2ca5ef2d83615bc6d21989a316c1bf19effe6585cdec86" exitCode=0
Oct 31 13:09:51.718084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:51.718005    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp" event={"ID":"aadd0aca-43fa-4851-8e0a-c0781bf55f32","Type":"ContainerDied","Data":"429f6b87443d71f9aa2ca5ef2d83615bc6d21989a316c1bf19effe6585cdec86"}
Oct 31 13:09:52.724637 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:52.724600    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp" event={"ID":"aadd0aca-43fa-4851-8e0a-c0781bf55f32","Type":"ContainerStarted","Data":"ae0f552ec4ec4c392624460383596f893db48f2fa894f742240bc930dc527faf"}
Oct 31 13:09:52.725063 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:52.724677    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:09:52.782020 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:52.781963    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp" podStartSLOduration=1.9166203689999999 podStartE2EDuration="5.781948229s" totalImagesPullingTime="3.305873352s" totalInitContainerRuntime="559.454508ms" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:52.78132243 +0000 UTC m=+392.399981591" watchObservedRunningTime="2026-10-31 13:09:52.781948229 +0000 UTC m=+392.400607393"
Oct 31 13:09:54.684286 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:54.684249    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm"]
Oct 31 13:09:54.687800 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:54.687779    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm"
Oct 31 13:09:54.698608 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:54.698581    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 31 13:09:54.698724 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:54.698688    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:09:54.698822 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:54.698809    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-krn79\""
Oct 31 13:09:54.703571 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:54.703546    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm"]
Oct 31 13:09:54.761472 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:54.761429    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c9x8\" (UniqueName: \"kubernetes.io/projected/69a32096-2eca-48e0-bb54-ba28d404bffc-kube-api-access-2c9x8\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm\" (UID: \"69a32096-2eca-48e0-bb54-ba28d404bffc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm"
Oct 31 13:09:54.761472 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:54.761472    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69a32096-2eca-48e0-bb54-ba28d404bffc-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm\" (UID: \"69a32096-2eca-48e0-bb54-ba28d404bffc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm"
Oct 31 13:09:54.862312 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:54.862255    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9x8\" (UniqueName: \"kubernetes.io/projected/69a32096-2eca-48e0-bb54-ba28d404bffc-kube-api-access-2c9x8\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm\" (UID: \"69a32096-2eca-48e0-bb54-ba28d404bffc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm"
Oct 31 13:09:54.862462 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:54.862321    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69a32096-2eca-48e0-bb54-ba28d404bffc-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm\" (UID: \"69a32096-2eca-48e0-bb54-ba28d404bffc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm"
Oct 31 13:09:54.862687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:54.862671    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69a32096-2eca-48e0-bb54-ba28d404bffc-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm\" (UID: \"69a32096-2eca-48e0-bb54-ba28d404bffc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm"
Oct 31 13:09:54.875737 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:54.875712    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c9x8\" (UniqueName: \"kubernetes.io/projected/69a32096-2eca-48e0-bb54-ba28d404bffc-kube-api-access-2c9x8\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm\" (UID: \"69a32096-2eca-48e0-bb54-ba28d404bffc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm"
Oct 31 13:09:54.999193 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:54.999102    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm"
Oct 31 13:09:55.287083 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:55.286968    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm"]
Oct 31 13:09:55.290011 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:09:55.289981    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a32096_2eca_48e0_bb54_ba28d404bffc.slice/crio-38319c8cc45ab5a6db2e268e34469b9f2f0a15adefb804a98916c35c1f44134d WatchSource:0}: Error finding container 38319c8cc45ab5a6db2e268e34469b9f2f0a15adefb804a98916c35c1f44134d: Status 404 returned error can't find the container with id 38319c8cc45ab5a6db2e268e34469b9f2f0a15adefb804a98916c35c1f44134d
Oct 31 13:09:55.743402 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:09:55.743361    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm" event={"ID":"69a32096-2eca-48e0-bb54-ba28d404bffc","Type":"ContainerStarted","Data":"38319c8cc45ab5a6db2e268e34469b9f2f0a15adefb804a98916c35c1f44134d"}
Oct 31 13:10:03.735214 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:03.734989    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-ldbbp"
Oct 31 13:10:03.950628 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:03.950589    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-zvfgt"]
Oct 31 13:10:03.954513 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:03.954487    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zvfgt"
Oct 31 13:10:03.959577 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:03.959538    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dfw4s\""
Oct 31 13:10:03.959577 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:03.959542    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 13:10:03.959735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:03.959577    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 13:10:03.966571 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:03.966548    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-zvfgt"]
Oct 31 13:10:04.039000 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:04.038923    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpxhj\" (UniqueName: \"kubernetes.io/projected/8b5c39b4-e8f2-45dc-8c45-3dd03ac475fa-kube-api-access-rpxhj\") pod \"limitador-operator-controller-manager-5778bb4449-zvfgt\" (UID: \"8b5c39b4-e8f2-45dc-8c45-3dd03ac475fa\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zvfgt"
Oct 31 13:10:04.139856 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:04.139818    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpxhj\" (UniqueName: \"kubernetes.io/projected/8b5c39b4-e8f2-45dc-8c45-3dd03ac475fa-kube-api-access-rpxhj\") pod \"limitador-operator-controller-manager-5778bb4449-zvfgt\" (UID: \"8b5c39b4-e8f2-45dc-8c45-3dd03ac475fa\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zvfgt"
Oct 31 13:10:04.153748 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:04.153717    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpxhj\" (UniqueName: \"kubernetes.io/projected/8b5c39b4-e8f2-45dc-8c45-3dd03ac475fa-kube-api-access-rpxhj\") pod \"limitador-operator-controller-manager-5778bb4449-zvfgt\" (UID: \"8b5c39b4-e8f2-45dc-8c45-3dd03ac475fa\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zvfgt"
Oct 31 13:10:04.271261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:04.271229    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zvfgt"
Oct 31 13:10:04.552205 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:04.552177    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-zvfgt"]
Oct 31 13:10:04.554765 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:10:04.554732    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5c39b4_e8f2_45dc_8c45_3dd03ac475fa.slice/crio-f268a744ade8a3499df7c2e39a72e2087e2aa256547a77a4a598b9c45c24c904 WatchSource:0}: Error finding container f268a744ade8a3499df7c2e39a72e2087e2aa256547a77a4a598b9c45c24c904: Status 404 returned error can't find the container with id f268a744ade8a3499df7c2e39a72e2087e2aa256547a77a4a598b9c45c24c904
Oct 31 13:10:04.789178 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:04.789142    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zvfgt" event={"ID":"8b5c39b4-e8f2-45dc-8c45-3dd03ac475fa","Type":"ContainerStarted","Data":"f268a744ade8a3499df7c2e39a72e2087e2aa256547a77a4a598b9c45c24c904"}
Oct 31 13:10:04.790836 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:04.790805    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm" event={"ID":"69a32096-2eca-48e0-bb54-ba28d404bffc","Type":"ContainerStarted","Data":"c35beb4eaedb2f1e96117aad79065d066379f8d88891b485b1f25c4d870bef71"}
Oct 31 13:10:04.817185 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:04.817133    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnmzm" podStartSLOduration=1.736787031 podStartE2EDuration="10.817116141s" totalImagesPullingTime="9.08032911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:04.815678128 +0000 UTC m=+404.434337292" watchObservedRunningTime="2026-10-31 13:10:04.817116141 +0000 UTC m=+404.435775305"
Oct 31 13:10:07.809266 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:07.809175    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zvfgt" event={"ID":"8b5c39b4-e8f2-45dc-8c45-3dd03ac475fa","Type":"ContainerStarted","Data":"884699a23c1cf43766b06955b97953b9cf3f86e9bc6837946b32c284b99c93ab"}
Oct 31 13:10:07.809676 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:07.809332    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zvfgt"
Oct 31 13:10:07.833718 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:07.833656    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zvfgt" podStartSLOduration=1.835961966 podStartE2EDuration="4.833641636s" totalImagesPullingTime="2.99767967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:07.83194788 +0000 UTC m=+407.450607067" watchObservedRunningTime="2026-10-31 13:10:07.833641636 +0000 UTC m=+407.452300799"
Oct 31 13:10:08.406651 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:08.406618    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9lrk5"]
Oct 31 13:10:08.410999 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:08.410965    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9lrk5"
Oct 31 13:10:08.417614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:08.417583    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 31 13:10:08.417743 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:08.417618    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4b87n\""
Oct 31 13:10:08.423005 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:08.422983    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9lrk5"]
Oct 31 13:10:08.484487 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:08.484452    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv2k6\" (UniqueName: \"kubernetes.io/projected/227ac746-4ef3-4dde-aaaa-ba5f7c0e89c0-kube-api-access-lv2k6\") pod \"dns-operator-controller-manager-8557cbdb84-9lrk5\" (UID: \"227ac746-4ef3-4dde-aaaa-ba5f7c0e89c0\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9lrk5"
Oct 31 13:10:08.585285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:08.585249    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv2k6\" (UniqueName: \"kubernetes.io/projected/227ac746-4ef3-4dde-aaaa-ba5f7c0e89c0-kube-api-access-lv2k6\") pod \"dns-operator-controller-manager-8557cbdb84-9lrk5\" (UID: \"227ac746-4ef3-4dde-aaaa-ba5f7c0e89c0\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9lrk5"
Oct 31 13:10:08.604435 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:08.604405    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv2k6\" (UniqueName: \"kubernetes.io/projected/227ac746-4ef3-4dde-aaaa-ba5f7c0e89c0-kube-api-access-lv2k6\") pod \"dns-operator-controller-manager-8557cbdb84-9lrk5\" (UID: \"227ac746-4ef3-4dde-aaaa-ba5f7c0e89c0\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9lrk5"
Oct 31 13:10:08.723997 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:08.723901    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9lrk5"
Oct 31 13:10:09.002149 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:09.002119    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9lrk5"]
Oct 31 13:10:09.003672 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:10:09.003645    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227ac746_4ef3_4dde_aaaa_ba5f7c0e89c0.slice/crio-6d99397122e7dfba0575db1987e04b8b01cfbc06c56d79764f73e738350008ca WatchSource:0}: Error finding container 6d99397122e7dfba0575db1987e04b8b01cfbc06c56d79764f73e738350008ca: Status 404 returned error can't find the container with id 6d99397122e7dfba0575db1987e04b8b01cfbc06c56d79764f73e738350008ca
Oct 31 13:10:09.820775 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:09.820738    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9lrk5" event={"ID":"227ac746-4ef3-4dde-aaaa-ba5f7c0e89c0","Type":"ContainerStarted","Data":"6d99397122e7dfba0575db1987e04b8b01cfbc06c56d79764f73e738350008ca"}
Oct 31 13:10:10.827592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:10.827551    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9lrk5" event={"ID":"227ac746-4ef3-4dde-aaaa-ba5f7c0e89c0","Type":"ContainerStarted","Data":"82cc36f1efd9d3abdca69553ec494030e61652d9da1c5807bab31980d8077e12"}
Oct 31 13:10:10.827973 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:10.827733    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9lrk5"
Oct 31 13:10:10.852451 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:10.852385    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9lrk5" podStartSLOduration=1.162410399 podStartE2EDuration="2.852368354s" totalImagesPullingTime="1.689957955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:10.851343018 +0000 UTC m=+410.470002182" watchObservedRunningTime="2026-10-31 13:10:10.852368354 +0000 UTC m=+410.471027519"
Oct 31 13:10:15.175600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:15.175565    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"]
Oct 31 13:10:15.179231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:15.179211    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"
Oct 31 13:10:15.183594 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:15.183568    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ztmv6\""
Oct 31 13:10:15.195013 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:15.194984    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"]
Oct 31 13:10:15.345523 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:15.345482    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfp55\" (UniqueName: \"kubernetes.io/projected/d4036bb0-42f3-4e30-b7d0-61f2bd470736-kube-api-access-wfp55\") pod \"kuadrant-operator-controller-manager-678d87766f-gc7w2\" (UID: \"d4036bb0-42f3-4e30-b7d0-61f2bd470736\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"
Oct 31 13:10:15.345736 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:15.345534    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4036bb0-42f3-4e30-b7d0-61f2bd470736-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gc7w2\" (UID: \"d4036bb0-42f3-4e30-b7d0-61f2bd470736\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"
Oct 31 13:10:15.446248 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:15.446154    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4036bb0-42f3-4e30-b7d0-61f2bd470736-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gc7w2\" (UID: \"d4036bb0-42f3-4e30-b7d0-61f2bd470736\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"
Oct 31 13:10:15.446423 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:15.446261    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfp55\" (UniqueName: \"kubernetes.io/projected/d4036bb0-42f3-4e30-b7d0-61f2bd470736-kube-api-access-wfp55\") pod \"kuadrant-operator-controller-manager-678d87766f-gc7w2\" (UID: \"d4036bb0-42f3-4e30-b7d0-61f2bd470736\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"
Oct 31 13:10:15.446600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:15.446579    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4036bb0-42f3-4e30-b7d0-61f2bd470736-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gc7w2\" (UID: \"d4036bb0-42f3-4e30-b7d0-61f2bd470736\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"
Oct 31 13:10:15.474923 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:15.474884    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfp55\" (UniqueName: \"kubernetes.io/projected/d4036bb0-42f3-4e30-b7d0-61f2bd470736-kube-api-access-wfp55\") pod \"kuadrant-operator-controller-manager-678d87766f-gc7w2\" (UID: \"d4036bb0-42f3-4e30-b7d0-61f2bd470736\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"
Oct 31 13:10:15.492213 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:15.492179    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"
Oct 31 13:10:15.781676 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:15.781649    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"]
Oct 31 13:10:15.783436 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:10:15.783407    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4036bb0_42f3_4e30_b7d0_61f2bd470736.slice/crio-ab04d8a3e1f745f6c063296d80df5df6080a77e415b462c4ac70a6f4518d6e24 WatchSource:0}: Error finding container ab04d8a3e1f745f6c063296d80df5df6080a77e415b462c4ac70a6f4518d6e24: Status 404 returned error can't find the container with id ab04d8a3e1f745f6c063296d80df5df6080a77e415b462c4ac70a6f4518d6e24
Oct 31 13:10:15.857103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:15.857057    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2" event={"ID":"d4036bb0-42f3-4e30-b7d0-61f2bd470736","Type":"ContainerStarted","Data":"ab04d8a3e1f745f6c063296d80df5df6080a77e415b462c4ac70a6f4518d6e24"}
Oct 31 13:10:18.818328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:18.818235    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zvfgt"
Oct 31 13:10:18.974158 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:18.974122    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"]
Oct 31 13:10:18.978091 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:18.978067    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:18.993901 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:18.993859    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-bbhmv\""
Oct 31 13:10:18.996541 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:18.996515    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"]
Oct 31 13:10:19.080728 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.080646    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdqwg\" (UniqueName: \"kubernetes.io/projected/35162b05-708e-4e60-8fe0-7d78a8ca24ae-kube-api-access-cdqwg\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.080920 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.080749    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35162b05-708e-4e60-8fe0-7d78a8ca24ae-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.080920 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.080798    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35162b05-708e-4e60-8fe0-7d78a8ca24ae-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.080920 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.080844    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35162b05-708e-4e60-8fe0-7d78a8ca24ae-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.081119 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.080995    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35162b05-708e-4e60-8fe0-7d78a8ca24ae-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.081119 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.081078    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35162b05-708e-4e60-8fe0-7d78a8ca24ae-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.081221 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.081122    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35162b05-708e-4e60-8fe0-7d78a8ca24ae-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.081221 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.081185    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35162b05-708e-4e60-8fe0-7d78a8ca24ae-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.081338 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.081239    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35162b05-708e-4e60-8fe0-7d78a8ca24ae-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.182474 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.182434    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdqwg\" (UniqueName: \"kubernetes.io/projected/35162b05-708e-4e60-8fe0-7d78a8ca24ae-kube-api-access-cdqwg\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.182474 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.182486    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35162b05-708e-4e60-8fe0-7d78a8ca24ae-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.182708 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.182521    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35162b05-708e-4e60-8fe0-7d78a8ca24ae-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.182708 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.182682    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35162b05-708e-4e60-8fe0-7d78a8ca24ae-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.182826 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.182753    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35162b05-708e-4e60-8fe0-7d78a8ca24ae-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.182826 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.182811    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35162b05-708e-4e60-8fe0-7d78a8ca24ae-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.182928 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.182843    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35162b05-708e-4e60-8fe0-7d78a8ca24ae-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.182928 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.182900    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35162b05-708e-4e60-8fe0-7d78a8ca24ae-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.183029 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.182925    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35162b05-708e-4e60-8fe0-7d78a8ca24ae-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.183207 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.183175    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35162b05-708e-4e60-8fe0-7d78a8ca24ae-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.183361 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.183252    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35162b05-708e-4e60-8fe0-7d78a8ca24ae-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.183790 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.183501    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35162b05-708e-4e60-8fe0-7d78a8ca24ae-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.183790 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.183573    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35162b05-708e-4e60-8fe0-7d78a8ca24ae-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.184031 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.184010    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35162b05-708e-4e60-8fe0-7d78a8ca24ae-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.185774 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.185748    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35162b05-708e-4e60-8fe0-7d78a8ca24ae-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.185985 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.185959    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35162b05-708e-4e60-8fe0-7d78a8ca24ae-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.196904 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.196870    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdqwg\" (UniqueName: \"kubernetes.io/projected/35162b05-708e-4e60-8fe0-7d78a8ca24ae-kube-api-access-cdqwg\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.197232 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.197187    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35162b05-708e-4e60-8fe0-7d78a8ca24ae-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf\" (UID: \"35162b05-708e-4e60-8fe0-7d78a8ca24ae\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.294085 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.294040    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:19.453321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.453271    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-grwkn"]
Oct 31 13:10:19.458014 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.457987    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-grwkn"
Oct 31 13:10:19.469208 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.469181    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-qx86t\""
Oct 31 13:10:19.470339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.470286    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-grwkn"]
Oct 31 13:10:19.586599 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.586564    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c95rg\" (UniqueName: \"kubernetes.io/projected/c831b6ce-162b-47af-a912-d53dfea59e2c-kube-api-access-c95rg\") pod \"authorino-operator-7b7bb4bc77-grwkn\" (UID: \"c831b6ce-162b-47af-a912-d53dfea59e2c\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-grwkn"
Oct 31 13:10:19.609590 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.609406    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"]
Oct 31 13:10:19.612514 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:10:19.612473    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35162b05_708e_4e60_8fe0_7d78a8ca24ae.slice/crio-71245c2bf330ba5a54a1a474da8f0287f433dc969187e83fdf5f2d1636468628 WatchSource:0}: Error finding container 71245c2bf330ba5a54a1a474da8f0287f433dc969187e83fdf5f2d1636468628: Status 404 returned error can't find the container with id 71245c2bf330ba5a54a1a474da8f0287f433dc969187e83fdf5f2d1636468628
Oct 31 13:10:19.615077 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.615044    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:10:19.615378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.615131    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:10:19.615378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.615171    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:10:19.687671 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.687641    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c95rg\" (UniqueName: \"kubernetes.io/projected/c831b6ce-162b-47af-a912-d53dfea59e2c-kube-api-access-c95rg\") pod \"authorino-operator-7b7bb4bc77-grwkn\" (UID: \"c831b6ce-162b-47af-a912-d53dfea59e2c\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-grwkn"
Oct 31 13:10:19.706550 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.706522    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c95rg\" (UniqueName: \"kubernetes.io/projected/c831b6ce-162b-47af-a912-d53dfea59e2c-kube-api-access-c95rg\") pod \"authorino-operator-7b7bb4bc77-grwkn\" (UID: \"c831b6ce-162b-47af-a912-d53dfea59e2c\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-grwkn"
Oct 31 13:10:19.776236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.775985    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-grwkn"
Oct 31 13:10:19.885384 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.885302    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf" event={"ID":"35162b05-708e-4e60-8fe0-7d78a8ca24ae","Type":"ContainerStarted","Data":"aa0af02db7bf8a19249bcb5e704f311ca3b02b61d502e322a11714f9ef17f1df"}
Oct 31 13:10:19.885384 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.885374    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf" event={"ID":"35162b05-708e-4e60-8fe0-7d78a8ca24ae","Type":"ContainerStarted","Data":"71245c2bf330ba5a54a1a474da8f0287f433dc969187e83fdf5f2d1636468628"}
Oct 31 13:10:19.913770 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:19.913706    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf" podStartSLOduration=1.91367712 podStartE2EDuration="1.91367712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:19.910496302 +0000 UTC m=+419.529155500" watchObservedRunningTime="2026-10-31 13:10:19.91367712 +0000 UTC m=+419.532336285"
Oct 31 13:10:20.073756 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:20.073732    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-grwkn"]
Oct 31 13:10:20.075196 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:10:20.075170    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc831b6ce_162b_47af_a912_d53dfea59e2c.slice/crio-c217491315b47b33b7a3ce2581c3013a92904c7cf75afb2218fffc4fcf17ed8f WatchSource:0}: Error finding container c217491315b47b33b7a3ce2581c3013a92904c7cf75afb2218fffc4fcf17ed8f: Status 404 returned error can't find the container with id c217491315b47b33b7a3ce2581c3013a92904c7cf75afb2218fffc4fcf17ed8f
Oct 31 13:10:20.294359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:20.294260    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:20.299133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:20.299106    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:20.732269 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:10:20.732227    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:10:20.892404 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:20.892372    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2" event={"ID":"d4036bb0-42f3-4e30-b7d0-61f2bd470736","Type":"ContainerStarted","Data":"d06590b7630913507fe950a5ef1b86ce6b45347b7258b8f1ba256c8c097b106f"}
Oct 31 13:10:20.892848 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:20.892488    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"
Oct 31 13:10:20.893872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:20.893849    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-grwkn" event={"ID":"c831b6ce-162b-47af-a912-d53dfea59e2c","Type":"ContainerStarted","Data":"c217491315b47b33b7a3ce2581c3013a92904c7cf75afb2218fffc4fcf17ed8f"}
Oct 31 13:10:20.894066 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:20.894051    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:20.895185 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:20.895159    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf"
Oct 31 13:10:20.926154 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:20.926080    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2" podStartSLOduration=1.672405367 podStartE2EDuration="5.926065792s" totalImagesPullingTime="4.253660425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:20.925112458 +0000 UTC m=+420.543771624" watchObservedRunningTime="2026-10-31 13:10:20.926065792 +0000 UTC m=+420.544724956"
Oct 31 13:10:21.838868 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:21.838839    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9lrk5"
Oct 31 13:10:22.391112 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.391078    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"]
Oct 31 13:10:22.402751 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.402724    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"
Oct 31 13:10:22.417207 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.417180    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-wnpkv\""
Oct 31 13:10:22.421756 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.421729    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"]
Oct 31 13:10:22.425949 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.425924    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 13:10:22.426088 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.425998    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 13:10:22.430735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.430711    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 31 13:10:22.513108 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.513066    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f5aeef9b-2b39-4e37-88ef-fadf328165d8-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-wr6rq\" (UID: \"f5aeef9b-2b39-4e37-88ef-fadf328165d8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"
Oct 31 13:10:22.513108 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.513107    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5aeef9b-2b39-4e37-88ef-fadf328165d8-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-wr6rq\" (UID: \"f5aeef9b-2b39-4e37-88ef-fadf328165d8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"
Oct 31 13:10:22.513329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.513206    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldsdj\" (UniqueName: \"kubernetes.io/projected/f5aeef9b-2b39-4e37-88ef-fadf328165d8-kube-api-access-ldsdj\") pod \"kserve-module-controller-manager-7566cf5fc8-wr6rq\" (UID: \"f5aeef9b-2b39-4e37-88ef-fadf328165d8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"
Oct 31 13:10:22.513329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.513240    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5aeef9b-2b39-4e37-88ef-fadf328165d8-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-wr6rq\" (UID: \"f5aeef9b-2b39-4e37-88ef-fadf328165d8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"
Oct 31 13:10:22.614079 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.614041    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5aeef9b-2b39-4e37-88ef-fadf328165d8-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-wr6rq\" (UID: \"f5aeef9b-2b39-4e37-88ef-fadf328165d8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"
Oct 31 13:10:22.614289 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.614267    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f5aeef9b-2b39-4e37-88ef-fadf328165d8-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-wr6rq\" (UID: \"f5aeef9b-2b39-4e37-88ef-fadf328165d8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"
Oct 31 13:10:22.614384 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.614317    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5aeef9b-2b39-4e37-88ef-fadf328165d8-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-wr6rq\" (UID: \"f5aeef9b-2b39-4e37-88ef-fadf328165d8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"
Oct 31 13:10:22.614384 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.614360    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldsdj\" (UniqueName: \"kubernetes.io/projected/f5aeef9b-2b39-4e37-88ef-fadf328165d8-kube-api-access-ldsdj\") pod \"kserve-module-controller-manager-7566cf5fc8-wr6rq\" (UID: \"f5aeef9b-2b39-4e37-88ef-fadf328165d8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"
Oct 31 13:10:22.614759 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.614736    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f5aeef9b-2b39-4e37-88ef-fadf328165d8-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-wr6rq\" (UID: \"f5aeef9b-2b39-4e37-88ef-fadf328165d8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"
Oct 31 13:10:22.614849 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.614758    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5aeef9b-2b39-4e37-88ef-fadf328165d8-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-wr6rq\" (UID: \"f5aeef9b-2b39-4e37-88ef-fadf328165d8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"
Oct 31 13:10:22.615136 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.615113    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5aeef9b-2b39-4e37-88ef-fadf328165d8-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-wr6rq\" (UID: \"f5aeef9b-2b39-4e37-88ef-fadf328165d8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"
Oct 31 13:10:22.636097 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.636058    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldsdj\" (UniqueName: \"kubernetes.io/projected/f5aeef9b-2b39-4e37-88ef-fadf328165d8-kube-api-access-ldsdj\") pod \"kserve-module-controller-manager-7566cf5fc8-wr6rq\" (UID: \"f5aeef9b-2b39-4e37-88ef-fadf328165d8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"
Oct 31 13:10:22.717153 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.717069    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"
Oct 31 13:10:22.907557 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.907517    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-grwkn" event={"ID":"c831b6ce-162b-47af-a912-d53dfea59e2c","Type":"ContainerStarted","Data":"b5d49105fb19bbaf26953935fcf22faa58417ee84d82858c293c4c33d63d17aa"}
Oct 31 13:10:22.907787 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.907767    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-grwkn"
Oct 31 13:10:22.932956 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:22.932902    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-grwkn" podStartSLOduration=2.14830518 podStartE2EDuration="3.932885908s" totalImagesPullingTime="1.784580728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:22.931075018 +0000 UTC m=+422.549734182" watchObservedRunningTime="2026-10-31 13:10:22.932885908 +0000 UTC m=+422.551545079"
Oct 31 13:10:23.001282 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:23.001256    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"]
Oct 31 13:10:23.002988 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:10:23.002957    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5aeef9b_2b39_4e37_88ef_fadf328165d8.slice/crio-3344e64fdf5079f1067171f84cf4eae1eb9397cd38597b0edac376bf64be2b30 WatchSource:0}: Error finding container 3344e64fdf5079f1067171f84cf4eae1eb9397cd38597b0edac376bf64be2b30: Status 404 returned error can't find the container with id 3344e64fdf5079f1067171f84cf4eae1eb9397cd38597b0edac376bf64be2b30
Oct 31 13:10:23.915526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:23.915486    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq" event={"ID":"f5aeef9b-2b39-4e37-88ef-fadf328165d8","Type":"ContainerStarted","Data":"3344e64fdf5079f1067171f84cf4eae1eb9397cd38597b0edac376bf64be2b30"}
Oct 31 13:10:25.927345 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:25.927283    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq" event={"ID":"f5aeef9b-2b39-4e37-88ef-fadf328165d8","Type":"ContainerStarted","Data":"8d5c3fe1101421e88f2226ba271d15181f36216c982cca6c58f1c5220727f64c"}
Oct 31 13:10:25.927745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:25.927376    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"
Oct 31 13:10:25.952911 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:25.952860    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq" podStartSLOduration=1.490859446 podStartE2EDuration="3.952846291s" totalImagesPullingTime="2.461986845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:25.950147334 +0000 UTC m=+425.568806497" watchObservedRunningTime="2026-10-31 13:10:25.952846291 +0000 UTC m=+425.571505455"
Oct 31 13:10:31.901738 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:31.901705    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"
Oct 31 13:10:33.918412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:33.918379    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-grwkn"
Oct 31 13:10:46.936564 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:10:46.936529    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wr6rq"
Oct 31 13:10:50.730748 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:10:50.730711    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:11:16.527090 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:16.527060    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"]
Oct 31 13:11:16.533485 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:16.533463    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"
Oct 31 13:11:16.544610 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:16.544585    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"]
Oct 31 13:11:16.556497 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:16.556469    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 31 13:11:16.556497 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:16.556478    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-cflm2\""
Oct 31 13:11:16.679182 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:16.679148    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr9f2\" (UniqueName: \"kubernetes.io/projected/d0f325ef-1db3-4475-a977-5af8cb1b10f7-kube-api-access-pr9f2\") pod \"llmisvc-controller-manager-869d5dcc5d-2vtnw\" (UID: \"d0f325ef-1db3-4475-a977-5af8cb1b10f7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"
Oct 31 13:11:16.679397 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:16.679269    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0f325ef-1db3-4475-a977-5af8cb1b10f7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2vtnw\" (UID: \"d0f325ef-1db3-4475-a977-5af8cb1b10f7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"
Oct 31 13:11:16.780840 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:16.780744    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0f325ef-1db3-4475-a977-5af8cb1b10f7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2vtnw\" (UID: \"d0f325ef-1db3-4475-a977-5af8cb1b10f7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"
Oct 31 13:11:16.781001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:16.780876    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr9f2\" (UniqueName: \"kubernetes.io/projected/d0f325ef-1db3-4475-a977-5af8cb1b10f7-kube-api-access-pr9f2\") pod \"llmisvc-controller-manager-869d5dcc5d-2vtnw\" (UID: \"d0f325ef-1db3-4475-a977-5af8cb1b10f7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"
Oct 31 13:11:16.783177 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:16.783149    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0f325ef-1db3-4475-a977-5af8cb1b10f7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2vtnw\" (UID: \"d0f325ef-1db3-4475-a977-5af8cb1b10f7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"
Oct 31 13:11:16.796071 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:16.796044    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr9f2\" (UniqueName: \"kubernetes.io/projected/d0f325ef-1db3-4475-a977-5af8cb1b10f7-kube-api-access-pr9f2\") pod \"llmisvc-controller-manager-869d5dcc5d-2vtnw\" (UID: \"d0f325ef-1db3-4475-a977-5af8cb1b10f7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"
Oct 31 13:11:16.847600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:16.847567    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"
Oct 31 13:11:17.196446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:17.196421    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"]
Oct 31 13:11:17.196553 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:11:17.196480    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd0f325ef_1db3_4475_a977_5af8cb1b10f7.slice/crio-c612b96956f529779e10884b7814a5fd726d1953985a43f8a4a2fb2fc729b293 WatchSource:0}: Error finding container c612b96956f529779e10884b7814a5fd726d1953985a43f8a4a2fb2fc729b293: Status 404 returned error can't find the container with id c612b96956f529779e10884b7814a5fd726d1953985a43f8a4a2fb2fc729b293
Oct 31 13:11:18.214758 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:18.214714    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw" event={"ID":"d0f325ef-1db3-4475-a977-5af8cb1b10f7","Type":"ContainerStarted","Data":"c612b96956f529779e10884b7814a5fd726d1953985a43f8a4a2fb2fc729b293"}
Oct 31 13:11:19.221688 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:19.221651    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw" event={"ID":"d0f325ef-1db3-4475-a977-5af8cb1b10f7","Type":"ContainerStarted","Data":"0175c639082d9b3ad4f2615c4c3520ce8985e0883fc3d4128c2e9e7d0a8d0d77"}
Oct 31 13:11:19.222121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:19.221777    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"
Oct 31 13:11:19.243161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:19.243063    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw" podStartSLOduration=1.451317267 podStartE2EDuration="3.243051882s" totalImagesPullingTime="1.791734615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:19.242180262 +0000 UTC m=+478.860839427" watchObservedRunningTime="2026-10-31 13:11:19.243051882 +0000 UTC m=+478.861711083"
Oct 31 13:11:20.731271 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:11:20.731233    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:11:50.229994 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:11:50.229917    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"
Oct 31 13:11:50.730704 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:11:50.730664    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:12:20.732017 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:12:20.731975    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:12:50.731039 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:12:50.730935    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:13:20.570730 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:13:20.570701    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:13:20.571768 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:13:20.571743    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:13:20.576401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:13:20.576381    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:13:20.577542 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:13:20.577520    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:13:20.588964 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:13:20.588933    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:13:20.590340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:13:20.590317    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:13:20.731091 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:13:20.731056    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:13:50.731452 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:13:50.731413    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:14:20.731411 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:14:20.731370    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:14:50.730809 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:14:50.730730    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:15:20.730946 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:15:20.730900    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:15:50.730590 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:15:50.730552    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:16:20.731337 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:16:20.731233    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:16:50.731078 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:16:50.731037    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:17:20.730664 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:17:20.730626    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:17:50.731255 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:17:50.731148    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:18:20.612858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:18:20.612829    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:18:20.615500 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:18:20.615475    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:18:20.618164 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:18:20.618145    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:18:20.621240 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:18:20.621221    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:18:20.626599 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:18:20.626580    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:18:20.629166 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:18:20.629141    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:18:20.730302 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:18:20.730253    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:18:50.730540 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:18:50.730505    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:19:20.731247 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:19:20.731169    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:19:50.730962 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:19:50.730926    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:20:20.730810 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:20:20.730765    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:20:38.687679 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:38.687596    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-gc7w2_d4036bb0-42f3-4e30-b7d0-61f2bd470736/manager/0.log"
Oct 31 13:20:38.875021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:38.874984    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"]
Oct 31 13:20:38.875209 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:38.875188    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2" podUID="d4036bb0-42f3-4e30-b7d0-61f2bd470736" containerName="manager" containerID="cri-o://d06590b7630913507fe950a5ef1b86ce6b45347b7258b8f1ba256c8c097b106f" gracePeriod=10
Oct 31 13:20:38.909857 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:38.909829    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"]
Oct 31 13:20:38.913411 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:38.913391    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"
Oct 31 13:20:38.926818 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:38.926791    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"]
Oct 31 13:20:38.960009 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:38.959986    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9w6f\" (UniqueName: \"kubernetes.io/projected/479e3854-ed33-4732-a369-d7d367ba245d-kube-api-access-p9w6f\") pod \"kuadrant-operator-controller-manager-678d87766f-9xc4x\" (UID: \"479e3854-ed33-4732-a369-d7d367ba245d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"
Oct 31 13:20:38.960129 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:38.960032    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/479e3854-ed33-4732-a369-d7d367ba245d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-9xc4x\" (UID: \"479e3854-ed33-4732-a369-d7d367ba245d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"
Oct 31 13:20:39.060922 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.060882    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9w6f\" (UniqueName: \"kubernetes.io/projected/479e3854-ed33-4732-a369-d7d367ba245d-kube-api-access-p9w6f\") pod \"kuadrant-operator-controller-manager-678d87766f-9xc4x\" (UID: \"479e3854-ed33-4732-a369-d7d367ba245d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"
Oct 31 13:20:39.061096 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.060966    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/479e3854-ed33-4732-a369-d7d367ba245d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-9xc4x\" (UID: \"479e3854-ed33-4732-a369-d7d367ba245d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"
Oct 31 13:20:39.061553 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.061487    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/479e3854-ed33-4732-a369-d7d367ba245d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-9xc4x\" (UID: \"479e3854-ed33-4732-a369-d7d367ba245d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"
Oct 31 13:20:39.074476 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.074448    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9w6f\" (UniqueName: \"kubernetes.io/projected/479e3854-ed33-4732-a369-d7d367ba245d-kube-api-access-p9w6f\") pod \"kuadrant-operator-controller-manager-678d87766f-9xc4x\" (UID: \"479e3854-ed33-4732-a369-d7d367ba245d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"
Oct 31 13:20:39.130971 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.130945    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"
Oct 31 13:20:39.131747 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.131721    4660 generic.go:356] "Generic (PLEG): container finished" podID="d4036bb0-42f3-4e30-b7d0-61f2bd470736" containerID="d06590b7630913507fe950a5ef1b86ce6b45347b7258b8f1ba256c8c097b106f" exitCode=0
Oct 31 13:20:39.131858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.131789    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2" event={"ID":"d4036bb0-42f3-4e30-b7d0-61f2bd470736","Type":"ContainerDied","Data":"d06590b7630913507fe950a5ef1b86ce6b45347b7258b8f1ba256c8c097b106f"}
Oct 31 13:20:39.131858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.131836    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2" event={"ID":"d4036bb0-42f3-4e30-b7d0-61f2bd470736","Type":"ContainerDied","Data":"ab04d8a3e1f745f6c063296d80df5df6080a77e415b462c4ac70a6f4518d6e24"}
Oct 31 13:20:39.131858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.131857    4660 scope.go:122] "RemoveContainer" containerID="d06590b7630913507fe950a5ef1b86ce6b45347b7258b8f1ba256c8c097b106f"
Oct 31 13:20:39.140695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.140677    4660 scope.go:122] "RemoveContainer" containerID="d06590b7630913507fe950a5ef1b86ce6b45347b7258b8f1ba256c8c097b106f"
Oct 31 13:20:39.140983 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:20:39.140964    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d06590b7630913507fe950a5ef1b86ce6b45347b7258b8f1ba256c8c097b106f\": container with ID starting with d06590b7630913507fe950a5ef1b86ce6b45347b7258b8f1ba256c8c097b106f not found: ID does not exist" containerID="d06590b7630913507fe950a5ef1b86ce6b45347b7258b8f1ba256c8c097b106f"
Oct 31 13:20:39.141044 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.140997    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d06590b7630913507fe950a5ef1b86ce6b45347b7258b8f1ba256c8c097b106f"} err="failed to get container status \"d06590b7630913507fe950a5ef1b86ce6b45347b7258b8f1ba256c8c097b106f\": rpc error: code = NotFound desc = could not find container \"d06590b7630913507fe950a5ef1b86ce6b45347b7258b8f1ba256c8c097b106f\": container with ID starting with d06590b7630913507fe950a5ef1b86ce6b45347b7258b8f1ba256c8c097b106f not found: ID does not exist"
Oct 31 13:20:39.162090 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.162060    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4036bb0-42f3-4e30-b7d0-61f2bd470736-kube-api-access-wfp55\" (UniqueName: \"kubernetes.io/projected/d4036bb0-42f3-4e30-b7d0-61f2bd470736-kube-api-access-wfp55\") pod \"d4036bb0-42f3-4e30-b7d0-61f2bd470736\" (UID: \"d4036bb0-42f3-4e30-b7d0-61f2bd470736\") "
Oct 31 13:20:39.162219 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.162112    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4036bb0-42f3-4e30-b7d0-61f2bd470736-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4036bb0-42f3-4e30-b7d0-61f2bd470736-extensions-socket-volume\") pod \"d4036bb0-42f3-4e30-b7d0-61f2bd470736\" (UID: \"d4036bb0-42f3-4e30-b7d0-61f2bd470736\") "
Oct 31 13:20:39.162722 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.162682    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4036bb0-42f3-4e30-b7d0-61f2bd470736-extensions-socket-volume" pod "d4036bb0-42f3-4e30-b7d0-61f2bd470736" (UID: "d4036bb0-42f3-4e30-b7d0-61f2bd470736"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:20:39.164688 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.164663    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4036bb0-42f3-4e30-b7d0-61f2bd470736-kube-api-access-wfp55" pod "d4036bb0-42f3-4e30-b7d0-61f2bd470736" (UID: "d4036bb0-42f3-4e30-b7d0-61f2bd470736"). InnerVolumeSpecName "kube-api-access-wfp55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:20:39.247143 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.247045    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"
Oct 31 13:20:39.263716 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.263679    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfp55\" (UniqueName: \"kubernetes.io/projected/d4036bb0-42f3-4e30-b7d0-61f2bd470736-kube-api-access-wfp55\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:20:39.263827 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.263719    4660 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4036bb0-42f3-4e30-b7d0-61f2bd470736-extensions-socket-volume\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:20:39.532578 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:39.532546    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"]
Oct 31 13:20:39.533362 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:20:39.533338    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479e3854_ed33_4732_a369_d7d367ba245d.slice/crio-16e8793ab85bc57d81a8e884bf844907fe4da128985947e487500ca33e4b3a70 WatchSource:0}: Error finding container 16e8793ab85bc57d81a8e884bf844907fe4da128985947e487500ca33e4b3a70: Status 404 returned error can't find the container with id 16e8793ab85bc57d81a8e884bf844907fe4da128985947e487500ca33e4b3a70
Oct 31 13:20:40.138057 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:40.138023    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"
Oct 31 13:20:40.139846 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:40.139821    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x" event={"ID":"479e3854-ed33-4732-a369-d7d367ba245d","Type":"ContainerStarted","Data":"b9a6560dd97bdb7b86e40f6d54285d619c183fd78d34e7e0310ef874af3e1755"}
Oct 31 13:20:40.139957 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:40.139863    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x" event={"ID":"479e3854-ed33-4732-a369-d7d367ba245d","Type":"ContainerStarted","Data":"16e8793ab85bc57d81a8e884bf844907fe4da128985947e487500ca33e4b3a70"}
Oct 31 13:20:40.139957 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:40.139884    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"
Oct 31 13:20:40.175733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:40.175690    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x" podStartSLOduration=2.175678111 podStartE2EDuration="2.175678111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:20:40.174818547 +0000 UTC m=+1039.793477711" watchObservedRunningTime="2026-10-31 13:20:40.175678111 +0000 UTC m=+1039.794337275"
Oct 31 13:20:40.194936 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:40.194908    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"]
Oct 31 13:20:40.200768 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:40.200737    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gc7w2"]
Oct 31 13:20:40.526571 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:40.526492    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4036bb0-42f3-4e30-b7d0-61f2bd470736" path="/var/lib/kubelet/pods/d4036bb0-42f3-4e30-b7d0-61f2bd470736/volumes"
Oct 31 13:20:50.730650 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:20:50.730616    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:20:51.147463 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:51.147433    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"
Oct 31 13:20:57.624553 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.624514    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7"]
Oct 31 13:20:57.625031 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.625012    4660 cpu_manager.go:414] "containerMap: removing container" podUID="d4036bb0-42f3-4e30-b7d0-61f2bd470736" containerName="manager"
Oct 31 13:20:57.625075 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.625035    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4036bb0-42f3-4e30-b7d0-61f2bd470736" containerName="manager"
Oct 31 13:20:57.625186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.625173    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4036bb0-42f3-4e30-b7d0-61f2bd470736" containerName="manager"
Oct 31 13:20:57.629640 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.629616    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7"
Oct 31 13:20:57.646515 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.646483    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7"]
Oct 31 13:20:57.646726 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.646698    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 31 13:20:57.646893 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.646824    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 31 13:20:57.646893 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.646889    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rmxc5\""
Oct 31 13:20:57.716663 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.716627    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8cc01303-cfd7-44cc-a013-50501b81c268-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-wl4g7\" (UID: \"8cc01303-cfd7-44cc-a013-50501b81c268\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7"
Oct 31 13:20:57.716663 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.716662    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cc01303-cfd7-44cc-a013-50501b81c268-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-wl4g7\" (UID: \"8cc01303-cfd7-44cc-a013-50501b81c268\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7"
Oct 31 13:20:57.716918 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.716695    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd46t\" (UniqueName: \"kubernetes.io/projected/8cc01303-cfd7-44cc-a013-50501b81c268-kube-api-access-sd46t\") pod \"kuadrant-console-plugin-66c4fb575d-wl4g7\" (UID: \"8cc01303-cfd7-44cc-a013-50501b81c268\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7"
Oct 31 13:20:57.817949 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.817913    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8cc01303-cfd7-44cc-a013-50501b81c268-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-wl4g7\" (UID: \"8cc01303-cfd7-44cc-a013-50501b81c268\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7"
Oct 31 13:20:57.817949 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.817949    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cc01303-cfd7-44cc-a013-50501b81c268-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-wl4g7\" (UID: \"8cc01303-cfd7-44cc-a013-50501b81c268\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7"
Oct 31 13:20:57.818214 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.817982    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd46t\" (UniqueName: \"kubernetes.io/projected/8cc01303-cfd7-44cc-a013-50501b81c268-kube-api-access-sd46t\") pod \"kuadrant-console-plugin-66c4fb575d-wl4g7\" (UID: \"8cc01303-cfd7-44cc-a013-50501b81c268\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7"
Oct 31 13:20:57.818682 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.818660    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8cc01303-cfd7-44cc-a013-50501b81c268-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-wl4g7\" (UID: \"8cc01303-cfd7-44cc-a013-50501b81c268\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7"
Oct 31 13:20:57.820355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.820332    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cc01303-cfd7-44cc-a013-50501b81c268-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-wl4g7\" (UID: \"8cc01303-cfd7-44cc-a013-50501b81c268\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7"
Oct 31 13:20:57.831538 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.831512    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd46t\" (UniqueName: \"kubernetes.io/projected/8cc01303-cfd7-44cc-a013-50501b81c268-kube-api-access-sd46t\") pod \"kuadrant-console-plugin-66c4fb575d-wl4g7\" (UID: \"8cc01303-cfd7-44cc-a013-50501b81c268\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7"
Oct 31 13:20:57.951975 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:57.951892    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7"
Oct 31 13:20:58.231462 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:58.231433    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7"]
Oct 31 13:20:58.232610 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:20:58.232582    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cc01303_cfd7_44cc_a013_50501b81c268.slice/crio-c653afb6b3b619b2976ab9b69fe7192496081933dc8fcdd99c8e2ac2c54896ac WatchSource:0}: Error finding container c653afb6b3b619b2976ab9b69fe7192496081933dc8fcdd99c8e2ac2c54896ac: Status 404 returned error can't find the container with id c653afb6b3b619b2976ab9b69fe7192496081933dc8fcdd99c8e2ac2c54896ac
Oct 31 13:20:58.233967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:58.233951    4660 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:20:58.393135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:58.393103    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-trhtj"]
Oct 31 13:20:58.397916 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:58.397897    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-trhtj"
Oct 31 13:20:58.404011 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:58.403979    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wp529\""
Oct 31 13:20:58.405496 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:58.405470    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-trhtj"]
Oct 31 13:20:58.524250 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:58.524161    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhdxg\" (UniqueName: \"kubernetes.io/projected/664ac3ad-f559-4f89-a61c-eb18c0318589-kube-api-access-rhdxg\") pod \"authorino-57f7f8659f-trhtj\" (UID: \"664ac3ad-f559-4f89-a61c-eb18c0318589\") " pod="kuadrant-system/authorino-57f7f8659f-trhtj"
Oct 31 13:20:58.625470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:58.625437    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhdxg\" (UniqueName: \"kubernetes.io/projected/664ac3ad-f559-4f89-a61c-eb18c0318589-kube-api-access-rhdxg\") pod \"authorino-57f7f8659f-trhtj\" (UID: \"664ac3ad-f559-4f89-a61c-eb18c0318589\") " pod="kuadrant-system/authorino-57f7f8659f-trhtj"
Oct 31 13:20:58.639078 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:58.639049    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhdxg\" (UniqueName: \"kubernetes.io/projected/664ac3ad-f559-4f89-a61c-eb18c0318589-kube-api-access-rhdxg\") pod \"authorino-57f7f8659f-trhtj\" (UID: \"664ac3ad-f559-4f89-a61c-eb18c0318589\") " pod="kuadrant-system/authorino-57f7f8659f-trhtj"
Oct 31 13:20:58.711529 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:58.711497    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-trhtj"
Oct 31 13:20:58.987695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:58.987666    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-trhtj"]
Oct 31 13:20:58.989232 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:20:58.989200    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod664ac3ad_f559_4f89_a61c_eb18c0318589.slice/crio-4703a9302a375e85dd5370d22ead3f0a737c549c2cf11c6e865fa34fe93d2f48 WatchSource:0}: Error finding container 4703a9302a375e85dd5370d22ead3f0a737c549c2cf11c6e865fa34fe93d2f48: Status 404 returned error can't find the container with id 4703a9302a375e85dd5370d22ead3f0a737c549c2cf11c6e865fa34fe93d2f48
Oct 31 13:20:59.240481 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:59.240403    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7" event={"ID":"8cc01303-cfd7-44cc-a013-50501b81c268","Type":"ContainerStarted","Data":"c653afb6b3b619b2976ab9b69fe7192496081933dc8fcdd99c8e2ac2c54896ac"}
Oct 31 13:20:59.241940 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:20:59.241905    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-trhtj" event={"ID":"664ac3ad-f559-4f89-a61c-eb18c0318589","Type":"ContainerStarted","Data":"4703a9302a375e85dd5370d22ead3f0a737c549c2cf11c6e865fa34fe93d2f48"}
Oct 31 13:21:02.265761 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:02.265707    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-trhtj" event={"ID":"664ac3ad-f559-4f89-a61c-eb18c0318589","Type":"ContainerStarted","Data":"23a23fa37b800e0f430083f4d93b3cfe5b5b4c3b748afba54ecbfb054393d696"}
Oct 31 13:21:02.283984 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:02.283931    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-trhtj" podStartSLOduration=1.958711905 podStartE2EDuration="4.283918064s" totalImagesPullingTime="2.325206159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:02.283635276 +0000 UTC m=+1061.902294441" watchObservedRunningTime="2026-10-31 13:21:02.283918064 +0000 UTC m=+1061.902577229"
Oct 31 13:21:02.701716 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:02.701672    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-trhtj"]
Oct 31 13:21:04.278309 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:04.278256    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7" event={"ID":"8cc01303-cfd7-44cc-a013-50501b81c268","Type":"ContainerStarted","Data":"d78d876e14ac64ca250253f614e4169d913a06c82356a096dae76ad1321c5a2f"}
Oct 31 13:21:04.278776 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:04.278389    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-trhtj" podUID="664ac3ad-f559-4f89-a61c-eb18c0318589" containerName="authorino" containerID="cri-o://23a23fa37b800e0f430083f4d93b3cfe5b5b4c3b748afba54ecbfb054393d696" gracePeriod=30
Oct 31 13:21:04.297631 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:04.297585    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wl4g7" podStartSLOduration=2.251756565 podStartE2EDuration="7.297572587s" totalImagesPullingTime="5.045816022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:20:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:04.296389142 +0000 UTC m=+1063.915048306" watchObservedRunningTime="2026-10-31 13:21:04.297572587 +0000 UTC m=+1063.916231751"
Oct 31 13:21:04.530425 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:04.530367    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-trhtj"
Oct 31 13:21:04.682869 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:04.682836    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/664ac3ad-f559-4f89-a61c-eb18c0318589-kube-api-access-rhdxg\" (UniqueName: \"kubernetes.io/projected/664ac3ad-f559-4f89-a61c-eb18c0318589-kube-api-access-rhdxg\") pod \"664ac3ad-f559-4f89-a61c-eb18c0318589\" (UID: \"664ac3ad-f559-4f89-a61c-eb18c0318589\") "
Oct 31 13:21:04.684955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:04.684920    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664ac3ad-f559-4f89-a61c-eb18c0318589-kube-api-access-rhdxg" pod "664ac3ad-f559-4f89-a61c-eb18c0318589" (UID: "664ac3ad-f559-4f89-a61c-eb18c0318589"). InnerVolumeSpecName "kube-api-access-rhdxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:21:04.784265 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:04.784185    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhdxg\" (UniqueName: \"kubernetes.io/projected/664ac3ad-f559-4f89-a61c-eb18c0318589-kube-api-access-rhdxg\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:21:05.284774 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:05.284737    4660 generic.go:356] "Generic (PLEG): container finished" podID="664ac3ad-f559-4f89-a61c-eb18c0318589" containerID="23a23fa37b800e0f430083f4d93b3cfe5b5b4c3b748afba54ecbfb054393d696" exitCode=0
Oct 31 13:21:05.285259 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:05.284785    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-trhtj"
Oct 31 13:21:05.285259 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:05.284821    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-trhtj" event={"ID":"664ac3ad-f559-4f89-a61c-eb18c0318589","Type":"ContainerDied","Data":"23a23fa37b800e0f430083f4d93b3cfe5b5b4c3b748afba54ecbfb054393d696"}
Oct 31 13:21:05.285259 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:05.284867    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-trhtj" event={"ID":"664ac3ad-f559-4f89-a61c-eb18c0318589","Type":"ContainerDied","Data":"4703a9302a375e85dd5370d22ead3f0a737c549c2cf11c6e865fa34fe93d2f48"}
Oct 31 13:21:05.285259 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:05.284887    4660 scope.go:122] "RemoveContainer" containerID="23a23fa37b800e0f430083f4d93b3cfe5b5b4c3b748afba54ecbfb054393d696"
Oct 31 13:21:05.294116 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:05.294093    4660 scope.go:122] "RemoveContainer" containerID="23a23fa37b800e0f430083f4d93b3cfe5b5b4c3b748afba54ecbfb054393d696"
Oct 31 13:21:05.294524 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:21:05.294505    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a23fa37b800e0f430083f4d93b3cfe5b5b4c3b748afba54ecbfb054393d696\": container with ID starting with 23a23fa37b800e0f430083f4d93b3cfe5b5b4c3b748afba54ecbfb054393d696 not found: ID does not exist" containerID="23a23fa37b800e0f430083f4d93b3cfe5b5b4c3b748afba54ecbfb054393d696"
Oct 31 13:21:05.294570 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:05.294534    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a23fa37b800e0f430083f4d93b3cfe5b5b4c3b748afba54ecbfb054393d696"} err="failed to get container status \"23a23fa37b800e0f430083f4d93b3cfe5b5b4c3b748afba54ecbfb054393d696\": rpc error: code = NotFound desc = could not find container \"23a23fa37b800e0f430083f4d93b3cfe5b5b4c3b748afba54ecbfb054393d696\": container with ID starting with 23a23fa37b800e0f430083f4d93b3cfe5b5b4c3b748afba54ecbfb054393d696 not found: ID does not exist"
Oct 31 13:21:05.307683 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:05.307649    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-trhtj"]
Oct 31 13:21:05.312005 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:05.311984    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-trhtj"]
Oct 31 13:21:06.525833 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:06.525793    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="664ac3ad-f559-4f89-a61c-eb18c0318589" path="/var/lib/kubelet/pods/664ac3ad-f559-4f89-a61c-eb18c0318589/volumes"
Oct 31 13:21:20.730555 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:21:20.730514    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:21:38.587821 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.587786    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9sxzh"]
Oct 31 13:21:38.588222 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.588144    4660 cpu_manager.go:414] "containerMap: removing container" podUID="664ac3ad-f559-4f89-a61c-eb18c0318589" containerName="authorino"
Oct 31 13:21:38.588222 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.588155    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="664ac3ad-f559-4f89-a61c-eb18c0318589" containerName="authorino"
Oct 31 13:21:38.588311 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.588258    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="664ac3ad-f559-4f89-a61c-eb18c0318589" containerName="authorino"
Oct 31 13:21:38.591188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.591171    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9sxzh"
Oct 31 13:21:38.595583 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.595552    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 13:21:38.595583 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.595567    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 13:21:38.596215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.596196    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 13:21:38.596256 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.596206    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zjpdd\""
Oct 31 13:21:38.605115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.605088    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9sxzh"]
Oct 31 13:21:38.669949 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.669919    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27a9de1e-24be-4171-b483-831e97275c78-webhook-cert\") pod \"maas-controller-579cf7c6d-9sxzh\" (UID: \"27a9de1e-24be-4171-b483-831e97275c78\") " pod="opendatahub/maas-controller-579cf7c6d-9sxzh"
Oct 31 13:21:38.669949 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.669953    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27a9de1e-24be-4171-b483-831e97275c78-metrics-certs\") pod \"maas-controller-579cf7c6d-9sxzh\" (UID: \"27a9de1e-24be-4171-b483-831e97275c78\") " pod="opendatahub/maas-controller-579cf7c6d-9sxzh"
Oct 31 13:21:38.670196 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.670023    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49t85\" (UniqueName: \"kubernetes.io/projected/27a9de1e-24be-4171-b483-831e97275c78-kube-api-access-49t85\") pod \"maas-controller-579cf7c6d-9sxzh\" (UID: \"27a9de1e-24be-4171-b483-831e97275c78\") " pod="opendatahub/maas-controller-579cf7c6d-9sxzh"
Oct 31 13:21:38.770836 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.770793    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27a9de1e-24be-4171-b483-831e97275c78-webhook-cert\") pod \"maas-controller-579cf7c6d-9sxzh\" (UID: \"27a9de1e-24be-4171-b483-831e97275c78\") " pod="opendatahub/maas-controller-579cf7c6d-9sxzh"
Oct 31 13:21:38.770836 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.770839    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27a9de1e-24be-4171-b483-831e97275c78-metrics-certs\") pod \"maas-controller-579cf7c6d-9sxzh\" (UID: \"27a9de1e-24be-4171-b483-831e97275c78\") " pod="opendatahub/maas-controller-579cf7c6d-9sxzh"
Oct 31 13:21:38.771094 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.770901    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49t85\" (UniqueName: \"kubernetes.io/projected/27a9de1e-24be-4171-b483-831e97275c78-kube-api-access-49t85\") pod \"maas-controller-579cf7c6d-9sxzh\" (UID: \"27a9de1e-24be-4171-b483-831e97275c78\") " pod="opendatahub/maas-controller-579cf7c6d-9sxzh"
Oct 31 13:21:38.773434 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.773404    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27a9de1e-24be-4171-b483-831e97275c78-webhook-cert\") pod \"maas-controller-579cf7c6d-9sxzh\" (UID: \"27a9de1e-24be-4171-b483-831e97275c78\") " pod="opendatahub/maas-controller-579cf7c6d-9sxzh"
Oct 31 13:21:38.773561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.773531    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27a9de1e-24be-4171-b483-831e97275c78-metrics-certs\") pod \"maas-controller-579cf7c6d-9sxzh\" (UID: \"27a9de1e-24be-4171-b483-831e97275c78\") " pod="opendatahub/maas-controller-579cf7c6d-9sxzh"
Oct 31 13:21:38.784736 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.784708    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49t85\" (UniqueName: \"kubernetes.io/projected/27a9de1e-24be-4171-b483-831e97275c78-kube-api-access-49t85\") pod \"maas-controller-579cf7c6d-9sxzh\" (UID: \"27a9de1e-24be-4171-b483-831e97275c78\") " pod="opendatahub/maas-controller-579cf7c6d-9sxzh"
Oct 31 13:21:38.897016 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.896979    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56949457b6-mx968"]
Oct 31 13:21:38.900698 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.900673    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56949457b6-mx968"
Oct 31 13:21:38.904575 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.904551    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9sxzh"
Oct 31 13:21:38.911818 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.911788    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56949457b6-mx968"]
Oct 31 13:21:38.972414 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.972387    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cf14527-ef89-4011-bad8-4ccd4bc245ec-metrics-certs\") pod \"maas-controller-56949457b6-mx968\" (UID: \"5cf14527-ef89-4011-bad8-4ccd4bc245ec\") " pod="opendatahub/maas-controller-56949457b6-mx968"
Oct 31 13:21:38.972549 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.972430    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cf14527-ef89-4011-bad8-4ccd4bc245ec-webhook-cert\") pod \"maas-controller-56949457b6-mx968\" (UID: \"5cf14527-ef89-4011-bad8-4ccd4bc245ec\") " pod="opendatahub/maas-controller-56949457b6-mx968"
Oct 31 13:21:38.972549 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:38.972530    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wvtm\" (UniqueName: \"kubernetes.io/projected/5cf14527-ef89-4011-bad8-4ccd4bc245ec-kube-api-access-9wvtm\") pod \"maas-controller-56949457b6-mx968\" (UID: \"5cf14527-ef89-4011-bad8-4ccd4bc245ec\") " pod="opendatahub/maas-controller-56949457b6-mx968"
Oct 31 13:21:39.073458 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:39.073427    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cf14527-ef89-4011-bad8-4ccd4bc245ec-metrics-certs\") pod \"maas-controller-56949457b6-mx968\" (UID: \"5cf14527-ef89-4011-bad8-4ccd4bc245ec\") " pod="opendatahub/maas-controller-56949457b6-mx968"
Oct 31 13:21:39.073609 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:39.073469    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cf14527-ef89-4011-bad8-4ccd4bc245ec-webhook-cert\") pod \"maas-controller-56949457b6-mx968\" (UID: \"5cf14527-ef89-4011-bad8-4ccd4bc245ec\") " pod="opendatahub/maas-controller-56949457b6-mx968"
Oct 31 13:21:39.073609 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:39.073518    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wvtm\" (UniqueName: \"kubernetes.io/projected/5cf14527-ef89-4011-bad8-4ccd4bc245ec-kube-api-access-9wvtm\") pod \"maas-controller-56949457b6-mx968\" (UID: \"5cf14527-ef89-4011-bad8-4ccd4bc245ec\") " pod="opendatahub/maas-controller-56949457b6-mx968"
Oct 31 13:21:39.075990 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:39.075956    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cf14527-ef89-4011-bad8-4ccd4bc245ec-webhook-cert\") pod \"maas-controller-56949457b6-mx968\" (UID: \"5cf14527-ef89-4011-bad8-4ccd4bc245ec\") " pod="opendatahub/maas-controller-56949457b6-mx968"
Oct 31 13:21:39.076107 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:39.075958    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cf14527-ef89-4011-bad8-4ccd4bc245ec-metrics-certs\") pod \"maas-controller-56949457b6-mx968\" (UID: \"5cf14527-ef89-4011-bad8-4ccd4bc245ec\") " pod="opendatahub/maas-controller-56949457b6-mx968"
Oct 31 13:21:39.087129 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:39.087106    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wvtm\" (UniqueName: \"kubernetes.io/projected/5cf14527-ef89-4011-bad8-4ccd4bc245ec-kube-api-access-9wvtm\") pod \"maas-controller-56949457b6-mx968\" (UID: \"5cf14527-ef89-4011-bad8-4ccd4bc245ec\") " pod="opendatahub/maas-controller-56949457b6-mx968"
Oct 31 13:21:39.190080 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:39.190051    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9sxzh"]
Oct 31 13:21:39.191938 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:21:39.191913    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a9de1e_24be_4171_b483_831e97275c78.slice/crio-87e0f10fce4554ff3fadbda9262b0d105c0790270787119fef3e05118286e102 WatchSource:0}: Error finding container 87e0f10fce4554ff3fadbda9262b0d105c0790270787119fef3e05118286e102: Status 404 returned error can't find the container with id 87e0f10fce4554ff3fadbda9262b0d105c0790270787119fef3e05118286e102
Oct 31 13:21:39.215693 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:39.215665    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56949457b6-mx968"
Oct 31 13:21:39.465036 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:39.464949    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9sxzh" event={"ID":"27a9de1e-24be-4171-b483-831e97275c78","Type":"ContainerStarted","Data":"87e0f10fce4554ff3fadbda9262b0d105c0790270787119fef3e05118286e102"}
Oct 31 13:21:39.493603 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:39.493578    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56949457b6-mx968"]
Oct 31 13:21:39.495271 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:21:39.495234    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf14527_ef89_4011_bad8_4ccd4bc245ec.slice/crio-0581c4af733602b6b1995043f99ea553eb12522fc60060dc0d8f09b433476637 WatchSource:0}: Error finding container 0581c4af733602b6b1995043f99ea553eb12522fc60060dc0d8f09b433476637: Status 404 returned error can't find the container with id 0581c4af733602b6b1995043f99ea553eb12522fc60060dc0d8f09b433476637
Oct 31 13:21:40.480729 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:40.480692    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56949457b6-mx968" event={"ID":"5cf14527-ef89-4011-bad8-4ccd4bc245ec","Type":"ContainerStarted","Data":"0581c4af733602b6b1995043f99ea553eb12522fc60060dc0d8f09b433476637"}
Oct 31 13:21:42.494154 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:42.494053    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9sxzh" event={"ID":"27a9de1e-24be-4171-b483-831e97275c78","Type":"ContainerStarted","Data":"e72d2881be06f207fc06ff80b03def869011cb16d4540f2b06febdd4c5f0daf0"}
Oct 31 13:21:42.494594 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:42.494253    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-9sxzh"
Oct 31 13:21:42.495691 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:42.495671    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56949457b6-mx968" event={"ID":"5cf14527-ef89-4011-bad8-4ccd4bc245ec","Type":"ContainerStarted","Data":"200bf4f0eab1c415e741b153bb49adf1cbd838c9a8c6d25f9af93eb96fd23f61"}
Oct 31 13:21:42.495774 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:42.495724    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56949457b6-mx968"
Oct 31 13:21:42.525379 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:42.525310    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-9sxzh" podStartSLOduration=1.7573182539999999 podStartE2EDuration="4.525281103s" totalImagesPullingTime="2.767962849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:42.524540319 +0000 UTC m=+1102.143199484" watchObservedRunningTime="2026-10-31 13:21:42.525281103 +0000 UTC m=+1102.143940267"
Oct 31 13:21:42.560566 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:42.560512    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-56949457b6-mx968" podStartSLOduration=1.827949158 podStartE2EDuration="4.560497852s" totalImagesPullingTime="2.732548694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:42.558392193 +0000 UTC m=+1102.177051357" watchObservedRunningTime="2026-10-31 13:21:42.560497852 +0000 UTC m=+1102.179157015"
Oct 31 13:21:50.730766 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:21:50.730730    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:21:53.503315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:53.503271    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-9sxzh"
Oct 31 13:21:53.503794 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:53.503441    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56949457b6-mx968"
Oct 31 13:21:53.579452 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:53.579414    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9sxzh"]
Oct 31 13:21:53.579673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:53.579625    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-9sxzh" podUID="27a9de1e-24be-4171-b483-831e97275c78" containerName="manager" containerID="cri-o://e72d2881be06f207fc06ff80b03def869011cb16d4540f2b06febdd4c5f0daf0" gracePeriod=30
Oct 31 13:21:53.830202 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:53.830175    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9sxzh"
Oct 31 13:21:53.911907 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:53.911874    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a9de1e-24be-4171-b483-831e97275c78-kube-api-access-49t85\" (UniqueName: \"kubernetes.io/projected/27a9de1e-24be-4171-b483-831e97275c78-kube-api-access-49t85\") pod \"27a9de1e-24be-4171-b483-831e97275c78\" (UID: \"27a9de1e-24be-4171-b483-831e97275c78\") "
Oct 31 13:21:53.912073 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:53.911917    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27a9de1e-24be-4171-b483-831e97275c78-webhook-cert\" (UniqueName: \"kubernetes.io/secret/27a9de1e-24be-4171-b483-831e97275c78-webhook-cert\") pod \"27a9de1e-24be-4171-b483-831e97275c78\" (UID: \"27a9de1e-24be-4171-b483-831e97275c78\") "
Oct 31 13:21:53.912073 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:53.911942    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27a9de1e-24be-4171-b483-831e97275c78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27a9de1e-24be-4171-b483-831e97275c78-metrics-certs\") pod \"27a9de1e-24be-4171-b483-831e97275c78\" (UID: \"27a9de1e-24be-4171-b483-831e97275c78\") "
Oct 31 13:21:53.914081 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:53.914052    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27a9de1e-24be-4171-b483-831e97275c78-metrics-certs" pod "27a9de1e-24be-4171-b483-831e97275c78" (UID: "27a9de1e-24be-4171-b483-831e97275c78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:21:53.914208 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:53.914098    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27a9de1e-24be-4171-b483-831e97275c78-webhook-cert" pod "27a9de1e-24be-4171-b483-831e97275c78" (UID: "27a9de1e-24be-4171-b483-831e97275c78"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:21:53.914208 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:53.914107    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a9de1e-24be-4171-b483-831e97275c78-kube-api-access-49t85" pod "27a9de1e-24be-4171-b483-831e97275c78" (UID: "27a9de1e-24be-4171-b483-831e97275c78"). InnerVolumeSpecName "kube-api-access-49t85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:21:54.012829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:54.012797    4660 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27a9de1e-24be-4171-b483-831e97275c78-webhook-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:21:54.012829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:54.012823    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27a9de1e-24be-4171-b483-831e97275c78-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:21:54.012829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:54.012833    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49t85\" (UniqueName: \"kubernetes.io/projected/27a9de1e-24be-4171-b483-831e97275c78-kube-api-access-49t85\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:21:54.560132 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:54.560100    4660 generic.go:356] "Generic (PLEG): container finished" podID="27a9de1e-24be-4171-b483-831e97275c78" containerID="e72d2881be06f207fc06ff80b03def869011cb16d4540f2b06febdd4c5f0daf0" exitCode=0
Oct 31 13:21:54.560521 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:54.560148    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9sxzh"
Oct 31 13:21:54.560521 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:54.560188    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9sxzh" event={"ID":"27a9de1e-24be-4171-b483-831e97275c78","Type":"ContainerDied","Data":"e72d2881be06f207fc06ff80b03def869011cb16d4540f2b06febdd4c5f0daf0"}
Oct 31 13:21:54.560521 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:54.560242    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9sxzh" event={"ID":"27a9de1e-24be-4171-b483-831e97275c78","Type":"ContainerDied","Data":"87e0f10fce4554ff3fadbda9262b0d105c0790270787119fef3e05118286e102"}
Oct 31 13:21:54.560521 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:54.560265    4660 scope.go:122] "RemoveContainer" containerID="e72d2881be06f207fc06ff80b03def869011cb16d4540f2b06febdd4c5f0daf0"
Oct 31 13:21:54.569891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:54.569875    4660 scope.go:122] "RemoveContainer" containerID="e72d2881be06f207fc06ff80b03def869011cb16d4540f2b06febdd4c5f0daf0"
Oct 31 13:21:54.570155 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:21:54.570127    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e72d2881be06f207fc06ff80b03def869011cb16d4540f2b06febdd4c5f0daf0\": container with ID starting with e72d2881be06f207fc06ff80b03def869011cb16d4540f2b06febdd4c5f0daf0 not found: ID does not exist" containerID="e72d2881be06f207fc06ff80b03def869011cb16d4540f2b06febdd4c5f0daf0"
Oct 31 13:21:54.570227 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:54.570156    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e72d2881be06f207fc06ff80b03def869011cb16d4540f2b06febdd4c5f0daf0"} err="failed to get container status \"e72d2881be06f207fc06ff80b03def869011cb16d4540f2b06febdd4c5f0daf0\": rpc error: code = NotFound desc = could not find container \"e72d2881be06f207fc06ff80b03def869011cb16d4540f2b06febdd4c5f0daf0\": container with ID starting with e72d2881be06f207fc06ff80b03def869011cb16d4540f2b06febdd4c5f0daf0 not found: ID does not exist"
Oct 31 13:21:54.580716 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:54.580688    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9sxzh"]
Oct 31 13:21:54.587100 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:54.587075    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9sxzh"]
Oct 31 13:21:56.526225 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:21:56.526189    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27a9de1e-24be-4171-b483-831e97275c78" path="/var/lib/kubelet/pods/27a9de1e-24be-4171-b483-831e97275c78/volumes"
Oct 31 13:22:14.359537 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.359459    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"]
Oct 31 13:22:14.359909 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.359832    4660 cpu_manager.go:414] "containerMap: removing container" podUID="27a9de1e-24be-4171-b483-831e97275c78" containerName="manager"
Oct 31 13:22:14.359909 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.359845    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a9de1e-24be-4171-b483-831e97275c78" containerName="manager"
Oct 31 13:22:14.359981 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.359914    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a9de1e-24be-4171-b483-831e97275c78" containerName="manager"
Oct 31 13:22:14.365216 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.365195    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:14.372801 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.372768    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:22:14.373200 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.373172    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 13:22:14.373322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.373287    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:22:14.373399 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.373359    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bx4t6\""
Oct 31 13:22:14.373452 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.373398    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 13:22:14.375531 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.375510    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"]
Oct 31 13:22:14.487118 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.487080    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbckm\" (UniqueName: \"kubernetes.io/projected/3d44a47b-e95b-4434-beb6-9d2236292d64-kube-api-access-mbckm\") pod \"maas-api-6d669c7fcd-d5rhh\" (UID: \"3d44a47b-e95b-4434-beb6-9d2236292d64\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:14.487284 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.487212    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d44a47b-e95b-4434-beb6-9d2236292d64-metrics-certs\") pod \"maas-api-6d669c7fcd-d5rhh\" (UID: \"3d44a47b-e95b-4434-beb6-9d2236292d64\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:14.487284 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.487266    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d44a47b-e95b-4434-beb6-9d2236292d64-maas-api-tls\") pod \"maas-api-6d669c7fcd-d5rhh\" (UID: \"3d44a47b-e95b-4434-beb6-9d2236292d64\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:14.487375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.487322    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d44a47b-e95b-4434-beb6-9d2236292d64-openshift-service-ca\") pod \"maas-api-6d669c7fcd-d5rhh\" (UID: \"3d44a47b-e95b-4434-beb6-9d2236292d64\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:14.588794 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.588757    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d44a47b-e95b-4434-beb6-9d2236292d64-openshift-service-ca\") pod \"maas-api-6d669c7fcd-d5rhh\" (UID: \"3d44a47b-e95b-4434-beb6-9d2236292d64\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:14.589011 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.588822    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbckm\" (UniqueName: \"kubernetes.io/projected/3d44a47b-e95b-4434-beb6-9d2236292d64-kube-api-access-mbckm\") pod \"maas-api-6d669c7fcd-d5rhh\" (UID: \"3d44a47b-e95b-4434-beb6-9d2236292d64\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:14.589011 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.588905    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d44a47b-e95b-4434-beb6-9d2236292d64-metrics-certs\") pod \"maas-api-6d669c7fcd-d5rhh\" (UID: \"3d44a47b-e95b-4434-beb6-9d2236292d64\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:14.589011 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.588956    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d44a47b-e95b-4434-beb6-9d2236292d64-maas-api-tls\") pod \"maas-api-6d669c7fcd-d5rhh\" (UID: \"3d44a47b-e95b-4434-beb6-9d2236292d64\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:14.589594 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.589558    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d44a47b-e95b-4434-beb6-9d2236292d64-openshift-service-ca\") pod \"maas-api-6d669c7fcd-d5rhh\" (UID: \"3d44a47b-e95b-4434-beb6-9d2236292d64\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:14.591450 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.591429    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d44a47b-e95b-4434-beb6-9d2236292d64-metrics-certs\") pod \"maas-api-6d669c7fcd-d5rhh\" (UID: \"3d44a47b-e95b-4434-beb6-9d2236292d64\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:14.591598 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.591582    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d44a47b-e95b-4434-beb6-9d2236292d64-maas-api-tls\") pod \"maas-api-6d669c7fcd-d5rhh\" (UID: \"3d44a47b-e95b-4434-beb6-9d2236292d64\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:14.604069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.604040    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbckm\" (UniqueName: \"kubernetes.io/projected/3d44a47b-e95b-4434-beb6-9d2236292d64-kube-api-access-mbckm\") pod \"maas-api-6d669c7fcd-d5rhh\" (UID: \"3d44a47b-e95b-4434-beb6-9d2236292d64\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:14.678437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.678408    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:14.957407 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:14.957375    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"]
Oct 31 13:22:14.959066 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:22:14.959041    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d44a47b_e95b_4434_beb6_9d2236292d64.slice/crio-26360394283e7dfd7890eaa8f79b9ac5a2d7e384c22e05c34fbdfcb7f21e542a WatchSource:0}: Error finding container 26360394283e7dfd7890eaa8f79b9ac5a2d7e384c22e05c34fbdfcb7f21e542a: Status 404 returned error can't find the container with id 26360394283e7dfd7890eaa8f79b9ac5a2d7e384c22e05c34fbdfcb7f21e542a
Oct 31 13:22:15.679783 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:15.679735    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh" event={"ID":"3d44a47b-e95b-4434-beb6-9d2236292d64","Type":"ContainerStarted","Data":"26360394283e7dfd7890eaa8f79b9ac5a2d7e384c22e05c34fbdfcb7f21e542a"}
Oct 31 13:22:16.686421 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:16.686324    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh" event={"ID":"3d44a47b-e95b-4434-beb6-9d2236292d64","Type":"ContainerStarted","Data":"ca65204d3529a07bcbec039a12215473901e5ccca1bd9f420535806458aa9458"}
Oct 31 13:22:16.686779 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:16.686567    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:16.717936 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:16.717868    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh" podStartSLOduration=1.279671203 podStartE2EDuration="2.717847016s" totalImagesPullingTime="1.438175813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:22:16.714787357 +0000 UTC m=+1136.333446523" watchObservedRunningTime="2026-10-31 13:22:16.717847016 +0000 UTC m=+1136.336506181"
Oct 31 13:22:20.752042 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:22:20.752006    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:22:22.699105 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:22.699074    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:23.438026 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.437992    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-97psd"]
Oct 31 13:22:23.442759 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.442739    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-97psd"
Oct 31 13:22:23.447507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.447480    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wp529\""
Oct 31 13:22:23.450648 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.450623    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-97psd"]
Oct 31 13:22:23.490582 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.490550    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dt8r\" (UniqueName: \"kubernetes.io/projected/923920e2-96aa-4fdf-ac4a-d42afd5f9c9c-kube-api-access-8dt8r\") pod \"authorino-6f6944bcdd-97psd\" (UID: \"923920e2-96aa-4fdf-ac4a-d42afd5f9c9c\") " pod="kuadrant-system/authorino-6f6944bcdd-97psd"
Oct 31 13:22:23.505217 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.505188    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-97psd"]
Oct 31 13:22:23.505424 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:22:23.505406    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-8dt8r], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-97psd" podUID="923920e2-96aa-4fdf-ac4a-d42afd5f9c9c"
Oct 31 13:22:23.591007 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.590967    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dt8r\" (UniqueName: \"kubernetes.io/projected/923920e2-96aa-4fdf-ac4a-d42afd5f9c9c-kube-api-access-8dt8r\") pod \"authorino-6f6944bcdd-97psd\" (UID: \"923920e2-96aa-4fdf-ac4a-d42afd5f9c9c\") " pod="kuadrant-system/authorino-6f6944bcdd-97psd"
Oct 31 13:22:23.606222 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.606187    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dt8r\" (UniqueName: \"kubernetes.io/projected/923920e2-96aa-4fdf-ac4a-d42afd5f9c9c-kube-api-access-8dt8r\") pod \"authorino-6f6944bcdd-97psd\" (UID: \"923920e2-96aa-4fdf-ac4a-d42afd5f9c9c\") " pod="kuadrant-system/authorino-6f6944bcdd-97psd"
Oct 31 13:22:23.625490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.625450    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"]
Oct 31 13:22:23.629859 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.629834    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 13:22:23.638358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.638331    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"]
Oct 31 13:22:23.692008 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.691929    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-maas-api-tls\") pod \"maas-api-6d475f7bf9-9cjft\" (UID: \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 13:22:23.692008 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.691965    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-openshift-service-ca\") pod \"maas-api-6d475f7bf9-9cjft\" (UID: \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 13:22:23.692179 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.692157    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnhqj\" (UniqueName: \"kubernetes.io/projected/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-kube-api-access-mnhqj\") pod \"maas-api-6d475f7bf9-9cjft\" (UID: \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 13:22:23.692224 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.692211    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-metrics-certs\") pod \"maas-api-6d475f7bf9-9cjft\" (UID: \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 13:22:23.756909 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.756878    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-97psd"
Oct 31 13:22:23.764157 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.764131    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-97psd"
Oct 31 13:22:23.793330 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.793282    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/923920e2-96aa-4fdf-ac4a-d42afd5f9c9c-kube-api-access-8dt8r\" (UniqueName: \"kubernetes.io/projected/923920e2-96aa-4fdf-ac4a-d42afd5f9c9c-kube-api-access-8dt8r\") pod \"923920e2-96aa-4fdf-ac4a-d42afd5f9c9c\" (UID: \"923920e2-96aa-4fdf-ac4a-d42afd5f9c9c\") "
Oct 31 13:22:23.793525 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.793510    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnhqj\" (UniqueName: \"kubernetes.io/projected/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-kube-api-access-mnhqj\") pod \"maas-api-6d475f7bf9-9cjft\" (UID: \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 13:22:23.793580 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.793547    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-metrics-certs\") pod \"maas-api-6d475f7bf9-9cjft\" (UID: \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 13:22:23.793623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.793585    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-maas-api-tls\") pod \"maas-api-6d475f7bf9-9cjft\" (UID: \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 13:22:23.793623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.793611    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-openshift-service-ca\") pod \"maas-api-6d475f7bf9-9cjft\" (UID: \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 13:22:23.794359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.794330    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-openshift-service-ca\") pod \"maas-api-6d475f7bf9-9cjft\" (UID: \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 13:22:23.795778 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.795750    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/923920e2-96aa-4fdf-ac4a-d42afd5f9c9c-kube-api-access-8dt8r" pod "923920e2-96aa-4fdf-ac4a-d42afd5f9c9c" (UID: "923920e2-96aa-4fdf-ac4a-d42afd5f9c9c"). InnerVolumeSpecName "kube-api-access-8dt8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:22:23.796232 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.796217    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-maas-api-tls\") pod \"maas-api-6d475f7bf9-9cjft\" (UID: \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 13:22:23.796391 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.796372    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-metrics-certs\") pod \"maas-api-6d475f7bf9-9cjft\" (UID: \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 13:22:23.810176 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.810148    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnhqj\" (UniqueName: \"kubernetes.io/projected/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-kube-api-access-mnhqj\") pod \"maas-api-6d475f7bf9-9cjft\" (UID: \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 13:22:23.894156 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.894123    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dt8r\" (UniqueName: \"kubernetes.io/projected/923920e2-96aa-4fdf-ac4a-d42afd5f9c9c-kube-api-access-8dt8r\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:22:23.947554 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:23.947483    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 13:22:24.137476 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:24.137431    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:22:24.137673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:24.137499    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9" podUID="ab9e59d3-2ac8-4381-98fa-22d3f25d682b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:22:24.224788 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:24.224733    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"]
Oct 31 13:22:24.227178 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:22:24.227151    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6966c7ae_5ec8_4d5a_9aba_be497296c5f5.slice/crio-06c8d66bf237dff786733f1e2fab404b06ccb6c0a760e9eff287be44720ce151 WatchSource:0}: Error finding container 06c8d66bf237dff786733f1e2fab404b06ccb6c0a760e9eff287be44720ce151: Status 404 returned error can't find the container with id 06c8d66bf237dff786733f1e2fab404b06ccb6c0a760e9eff287be44720ce151
Oct 31 13:22:24.764565 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:24.764531    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft" event={"ID":"6966c7ae-5ec8-4d5a-9aba-be497296c5f5","Type":"ContainerStarted","Data":"e0aa92963ffcde300e2a9394bc5b4a2c1ff4f58a46087e00fa44f3fe13238da2"}
Oct 31 13:22:24.765012 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:24.764593    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft" event={"ID":"6966c7ae-5ec8-4d5a-9aba-be497296c5f5","Type":"ContainerStarted","Data":"06c8d66bf237dff786733f1e2fab404b06ccb6c0a760e9eff287be44720ce151"}
Oct 31 13:22:24.765012 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:24.764622    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-97psd"
Oct 31 13:22:24.765012 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:24.764760    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 13:22:24.787976 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:24.787881    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft" podStartSLOduration=1.348349506 podStartE2EDuration="1.787868734s" totalImagesPullingTime="439.519228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:22:24.785458742 +0000 UTC m=+1144.404117907" watchObservedRunningTime="2026-10-31 13:22:24.787868734 +0000 UTC m=+1144.406527944"
Oct 31 13:22:24.813099 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:24.813074    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-97psd"]
Oct 31 13:22:24.824322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:24.822077    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-97psd"]
Oct 31 13:22:25.772058 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:25.772025    4660 generic.go:356] "Generic (PLEG): container finished" podID="ab9e59d3-2ac8-4381-98fa-22d3f25d682b" containerID="7f75e2cdf47ba65a60221b84d475060213db568023350d096f246c98205203fe" exitCode=0
Oct 31 13:22:25.772590 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:25.772098    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9" event={"ID":"ab9e59d3-2ac8-4381-98fa-22d3f25d682b","Type":"ContainerDied","Data":"7f75e2cdf47ba65a60221b84d475060213db568023350d096f246c98205203fe"}
Oct 31 13:22:25.772700 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:25.772683    4660 scope.go:122] "RemoveContainer" containerID="7f75e2cdf47ba65a60221b84d475060213db568023350d096f246c98205203fe"
Oct 31 13:22:25.773726 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:25.773697    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:22:25.773832 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:25.773781    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:22:25.773832 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:25.773814    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:22:26.532976 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:26.532942    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="923920e2-96aa-4fdf-ac4a-d42afd5f9c9c" path="/var/lib/kubelet/pods/923920e2-96aa-4fdf-ac4a-d42afd5f9c9c/volumes"
Oct 31 13:22:26.779176 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:26.779143    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9" event={"ID":"ab9e59d3-2ac8-4381-98fa-22d3f25d682b","Type":"ContainerStarted","Data":"9e312cf2242b4ac381c7bfd9ae7d75f9c6e6716c470c1910b7a193d04c822552"}
Oct 31 13:22:26.927591 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:26.927554    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-dk7st"]
Oct 31 13:22:26.932450 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:26.932424    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-dk7st"]
Oct 31 13:22:26.932586 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:26.932554    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dk7st"
Oct 31 13:22:27.031832 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:27.031797    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c508770-dcf5-4daf-b831-78f03a35e81b-cert\") pod \"llmisvc-controller-manager-6466c554d8-dk7st\" (UID: \"1c508770-dcf5-4daf-b831-78f03a35e81b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dk7st"
Oct 31 13:22:27.032028 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:27.031853    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mrph\" (UniqueName: \"kubernetes.io/projected/1c508770-dcf5-4daf-b831-78f03a35e81b-kube-api-access-9mrph\") pod \"llmisvc-controller-manager-6466c554d8-dk7st\" (UID: \"1c508770-dcf5-4daf-b831-78f03a35e81b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dk7st"
Oct 31 13:22:27.061397 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:27.061366    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:22:27.062080 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:27.062060    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 13:22:27.062129 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:27.062107    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9" podUID="ab9e59d3-2ac8-4381-98fa-22d3f25d682b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 13:22:27.133228 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:27.133188    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c508770-dcf5-4daf-b831-78f03a35e81b-cert\") pod \"llmisvc-controller-manager-6466c554d8-dk7st\" (UID: \"1c508770-dcf5-4daf-b831-78f03a35e81b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dk7st"
Oct 31 13:22:27.133479 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:27.133248    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mrph\" (UniqueName: \"kubernetes.io/projected/1c508770-dcf5-4daf-b831-78f03a35e81b-kube-api-access-9mrph\") pod \"llmisvc-controller-manager-6466c554d8-dk7st\" (UID: \"1c508770-dcf5-4daf-b831-78f03a35e81b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dk7st"
Oct 31 13:22:27.135810 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:27.135786    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c508770-dcf5-4daf-b831-78f03a35e81b-cert\") pod \"llmisvc-controller-manager-6466c554d8-dk7st\" (UID: \"1c508770-dcf5-4daf-b831-78f03a35e81b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dk7st"
Oct 31 13:22:27.146365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:27.146340    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mrph\" (UniqueName: \"kubernetes.io/projected/1c508770-dcf5-4daf-b831-78f03a35e81b-kube-api-access-9mrph\") pod \"llmisvc-controller-manager-6466c554d8-dk7st\" (UID: \"1c508770-dcf5-4daf-b831-78f03a35e81b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dk7st"
Oct 31 13:22:27.245837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:27.245755    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dk7st"
Oct 31 13:22:27.532564 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:27.532537    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-dk7st"]
Oct 31 13:22:27.533925 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:22:27.533896    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c508770_dcf5_4daf_b831_78f03a35e81b.slice/crio-27e9d112105bfd457721b946fd740d55ac21c8e02aef7941ee10bd5b124d9e8d WatchSource:0}: Error finding container 27e9d112105bfd457721b946fd740d55ac21c8e02aef7941ee10bd5b124d9e8d: Status 404 returned error can't find the container with id 27e9d112105bfd457721b946fd740d55ac21c8e02aef7941ee10bd5b124d9e8d
Oct 31 13:22:27.788629 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:27.788535    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dk7st" event={"ID":"1c508770-dcf5-4daf-b831-78f03a35e81b","Type":"ContainerStarted","Data":"27e9d112105bfd457721b946fd740d55ac21c8e02aef7941ee10bd5b124d9e8d"}
Oct 31 13:22:28.062394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:28.062320    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 13:22:28.062539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:28.062370    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9" podUID="ab9e59d3-2ac8-4381-98fa-22d3f25d682b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 13:22:28.795515 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:28.795476    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dk7st" event={"ID":"1c508770-dcf5-4daf-b831-78f03a35e81b","Type":"ContainerStarted","Data":"df76e0d3e5af1e9f943ccaac14da0d8d6260f3670925820d7c64ef55e7b9dbbd"}
Oct 31 13:22:28.795998 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:28.795627    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dk7st"
Oct 31 13:22:28.839052 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:28.838997    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dk7st" podStartSLOduration=2.423783788 podStartE2EDuration="2.838983589s" totalImagesPullingTime="415.199801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:22:28.838255216 +0000 UTC m=+1148.456914380" watchObservedRunningTime="2026-10-31 13:22:28.838983589 +0000 UTC m=+1148.457643155"
Oct 31 13:22:29.061908 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:29.061823    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 13:22:29.061908 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:29.061879    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9" podUID="ab9e59d3-2ac8-4381-98fa-22d3f25d682b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 13:22:30.061690 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:30.061649    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 13:22:30.062181 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:30.061709    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9" podUID="ab9e59d3-2ac8-4381-98fa-22d3f25d682b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 13:22:30.778161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:30.778133    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 13:22:30.853473 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:30.853432    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"]
Oct 31 13:22:30.853673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:30.853651    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh" podUID="3d44a47b-e95b-4434-beb6-9d2236292d64" containerName="maas-api" containerID="cri-o://ca65204d3529a07bcbec039a12215473901e5ccca1bd9f420535806458aa9458" gracePeriod=30
Oct 31 13:22:31.061554 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.061524    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 13:22:31.061692 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.061573    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9" podUID="ab9e59d3-2ac8-4381-98fa-22d3f25d682b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 13:22:31.133613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.133588    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:31.166899 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.166867    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d44a47b-e95b-4434-beb6-9d2236292d64-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d44a47b-e95b-4434-beb6-9d2236292d64-metrics-certs\") pod \"3d44a47b-e95b-4434-beb6-9d2236292d64\" (UID: \"3d44a47b-e95b-4434-beb6-9d2236292d64\") "
Oct 31 13:22:31.167048 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.166938    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d44a47b-e95b-4434-beb6-9d2236292d64-kube-api-access-mbckm\" (UniqueName: \"kubernetes.io/projected/3d44a47b-e95b-4434-beb6-9d2236292d64-kube-api-access-mbckm\") pod \"3d44a47b-e95b-4434-beb6-9d2236292d64\" (UID: \"3d44a47b-e95b-4434-beb6-9d2236292d64\") "
Oct 31 13:22:31.167048 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.166961    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d44a47b-e95b-4434-beb6-9d2236292d64-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d44a47b-e95b-4434-beb6-9d2236292d64-maas-api-tls\") pod \"3d44a47b-e95b-4434-beb6-9d2236292d64\" (UID: \"3d44a47b-e95b-4434-beb6-9d2236292d64\") "
Oct 31 13:22:31.167048 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.166997    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d44a47b-e95b-4434-beb6-9d2236292d64-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d44a47b-e95b-4434-beb6-9d2236292d64-openshift-service-ca\") pod \"3d44a47b-e95b-4434-beb6-9d2236292d64\" (UID: \"3d44a47b-e95b-4434-beb6-9d2236292d64\") "
Oct 31 13:22:31.167456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.167438    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d44a47b-e95b-4434-beb6-9d2236292d64-openshift-service-ca" pod "3d44a47b-e95b-4434-beb6-9d2236292d64" (UID: "3d44a47b-e95b-4434-beb6-9d2236292d64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:22:31.169687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.169651    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d44a47b-e95b-4434-beb6-9d2236292d64-kube-api-access-mbckm" pod "3d44a47b-e95b-4434-beb6-9d2236292d64" (UID: "3d44a47b-e95b-4434-beb6-9d2236292d64"). InnerVolumeSpecName "kube-api-access-mbckm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:22:31.169687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.169682    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d44a47b-e95b-4434-beb6-9d2236292d64-metrics-certs" pod "3d44a47b-e95b-4434-beb6-9d2236292d64" (UID: "3d44a47b-e95b-4434-beb6-9d2236292d64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:22:31.169845 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.169727    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d44a47b-e95b-4434-beb6-9d2236292d64-maas-api-tls" pod "3d44a47b-e95b-4434-beb6-9d2236292d64" (UID: "3d44a47b-e95b-4434-beb6-9d2236292d64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:22:31.268469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.268425    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d44a47b-e95b-4434-beb6-9d2236292d64-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:22:31.268469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.268459    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbckm\" (UniqueName: \"kubernetes.io/projected/3d44a47b-e95b-4434-beb6-9d2236292d64-kube-api-access-mbckm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:22:31.268469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.268469    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d44a47b-e95b-4434-beb6-9d2236292d64-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:22:31.268469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.268479    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d44a47b-e95b-4434-beb6-9d2236292d64-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:22:31.841016 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.840978    4660 generic.go:356] "Generic (PLEG): container finished" podID="3d44a47b-e95b-4434-beb6-9d2236292d64" containerID="ca65204d3529a07bcbec039a12215473901e5ccca1bd9f420535806458aa9458" exitCode=0
Oct 31 13:22:31.841204 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.841029    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"
Oct 31 13:22:31.841204 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.841066    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh" event={"ID":"3d44a47b-e95b-4434-beb6-9d2236292d64","Type":"ContainerDied","Data":"ca65204d3529a07bcbec039a12215473901e5ccca1bd9f420535806458aa9458"}
Oct 31 13:22:31.841204 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.841120    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh" event={"ID":"3d44a47b-e95b-4434-beb6-9d2236292d64","Type":"ContainerDied","Data":"26360394283e7dfd7890eaa8f79b9ac5a2d7e384c22e05c34fbdfcb7f21e542a"}
Oct 31 13:22:31.841204 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.841143    4660 scope.go:122] "RemoveContainer" containerID="ca65204d3529a07bcbec039a12215473901e5ccca1bd9f420535806458aa9458"
Oct 31 13:22:31.851356 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.851334    4660 scope.go:122] "RemoveContainer" containerID="ca65204d3529a07bcbec039a12215473901e5ccca1bd9f420535806458aa9458"
Oct 31 13:22:31.851645 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:22:31.851626    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca65204d3529a07bcbec039a12215473901e5ccca1bd9f420535806458aa9458\": container with ID starting with ca65204d3529a07bcbec039a12215473901e5ccca1bd9f420535806458aa9458 not found: ID does not exist" containerID="ca65204d3529a07bcbec039a12215473901e5ccca1bd9f420535806458aa9458"
Oct 31 13:22:31.851733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.851656    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca65204d3529a07bcbec039a12215473901e5ccca1bd9f420535806458aa9458"} err="failed to get container status \"ca65204d3529a07bcbec039a12215473901e5ccca1bd9f420535806458aa9458\": rpc error: code = NotFound desc = could not find container \"ca65204d3529a07bcbec039a12215473901e5ccca1bd9f420535806458aa9458\": container with ID starting with ca65204d3529a07bcbec039a12215473901e5ccca1bd9f420535806458aa9458 not found: ID does not exist"
Oct 31 13:22:31.867736 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.867694    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"]
Oct 31 13:22:31.871815 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:31.871792    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-d5rhh"]
Oct 31 13:22:32.065310 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:32.065271    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:22:32.065641 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:32.065624    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:22:32.066379 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:32.066362    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9"
Oct 31 13:22:32.525890 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:32.525859    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d44a47b-e95b-4434-beb6-9d2236292d64" path="/var/lib/kubelet/pods/3d44a47b-e95b-4434-beb6-9d2236292d64/volumes"
Oct 31 13:22:50.731192 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:22:50.731149    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:22:59.828738 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:59.828705    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dk7st"
Oct 31 13:22:59.881259 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:59.881225    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"]
Oct 31 13:22:59.881474 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:59.881446    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw" podUID="d0f325ef-1db3-4475-a977-5af8cb1b10f7" containerName="manager" containerID="cri-o://0175c639082d9b3ad4f2615c4c3520ce8985e0883fc3d4128c2e9e7d0a8d0d77" gracePeriod=30
Oct 31 13:22:59.997656 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:59.997629    4660 generic.go:356] "Generic (PLEG): container finished" podID="d0f325ef-1db3-4475-a977-5af8cb1b10f7" containerID="0175c639082d9b3ad4f2615c4c3520ce8985e0883fc3d4128c2e9e7d0a8d0d77" exitCode=0
Oct 31 13:22:59.997774 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:22:59.997705    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw" event={"ID":"d0f325ef-1db3-4475-a977-5af8cb1b10f7","Type":"ContainerDied","Data":"0175c639082d9b3ad4f2615c4c3520ce8985e0883fc3d4128c2e9e7d0a8d0d77"}
Oct 31 13:23:00.130519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:00.130495    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"
Oct 31 13:23:00.218647 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:00.218611    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f325ef-1db3-4475-a977-5af8cb1b10f7-kube-api-access-pr9f2\" (UniqueName: \"kubernetes.io/projected/d0f325ef-1db3-4475-a977-5af8cb1b10f7-kube-api-access-pr9f2\") pod \"d0f325ef-1db3-4475-a977-5af8cb1b10f7\" (UID: \"d0f325ef-1db3-4475-a977-5af8cb1b10f7\") "
Oct 31 13:23:00.218855 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:00.218724    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0f325ef-1db3-4475-a977-5af8cb1b10f7-cert\" (UniqueName: \"kubernetes.io/secret/d0f325ef-1db3-4475-a977-5af8cb1b10f7-cert\") pod \"d0f325ef-1db3-4475-a977-5af8cb1b10f7\" (UID: \"d0f325ef-1db3-4475-a977-5af8cb1b10f7\") "
Oct 31 13:23:00.220845 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:00.220818    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f325ef-1db3-4475-a977-5af8cb1b10f7-kube-api-access-pr9f2" pod "d0f325ef-1db3-4475-a977-5af8cb1b10f7" (UID: "d0f325ef-1db3-4475-a977-5af8cb1b10f7"). InnerVolumeSpecName "kube-api-access-pr9f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:23:00.220955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:00.220824    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0f325ef-1db3-4475-a977-5af8cb1b10f7-cert" pod "d0f325ef-1db3-4475-a977-5af8cb1b10f7" (UID: "d0f325ef-1db3-4475-a977-5af8cb1b10f7"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:23:00.319700 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:00.319660    4660 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0f325ef-1db3-4475-a977-5af8cb1b10f7-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:23:00.319700 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:00.319692    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr9f2\" (UniqueName: \"kubernetes.io/projected/d0f325ef-1db3-4475-a977-5af8cb1b10f7-kube-api-access-pr9f2\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:23:01.004876 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:01.004845    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"
Oct 31 13:23:01.004876 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:01.004856    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw" event={"ID":"d0f325ef-1db3-4475-a977-5af8cb1b10f7","Type":"ContainerDied","Data":"c612b96956f529779e10884b7814a5fd726d1953985a43f8a4a2fb2fc729b293"}
Oct 31 13:23:01.005426 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:01.004916    4660 scope.go:122] "RemoveContainer" containerID="0175c639082d9b3ad4f2615c4c3520ce8985e0883fc3d4128c2e9e7d0a8d0d77"
Oct 31 13:23:01.027382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:01.027356    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"]
Oct 31 13:23:01.038489 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:01.038467    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2vtnw"]
Oct 31 13:23:02.526084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:02.526048    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f325ef-1db3-4475-a977-5af8cb1b10f7" path="/var/lib/kubelet/pods/d0f325ef-1db3-4475-a977-5af8cb1b10f7/volumes"
Oct 31 13:23:08.739590 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.739554    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"]
Oct 31 13:23:08.739967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.739941    4660 cpu_manager.go:414] "containerMap: removing container" podUID="3d44a47b-e95b-4434-beb6-9d2236292d64" containerName="maas-api"
Oct 31 13:23:08.739967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.739952    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d44a47b-e95b-4434-beb6-9d2236292d64" containerName="maas-api"
Oct 31 13:23:08.739967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.739966    4660 cpu_manager.go:414] "containerMap: removing container" podUID="d0f325ef-1db3-4475-a977-5af8cb1b10f7" containerName="manager"
Oct 31 13:23:08.740073 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.739971    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f325ef-1db3-4475-a977-5af8cb1b10f7" containerName="manager"
Oct 31 13:23:08.740073 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.740038    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d44a47b-e95b-4434-beb6-9d2236292d64" containerName="maas-api"
Oct 31 13:23:08.740073 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.740046    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f325ef-1db3-4475-a977-5af8cb1b10f7" containerName="manager"
Oct 31 13:23:08.743173 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.743157    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.758361 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.758338    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"]
Oct 31 13:23:08.891548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.891504    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.891548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.891550    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhxmf\" (UniqueName: \"kubernetes.io/projected/344707b1-2bf5-4481-8522-afe6d996c26c-kube-api-access-hhxmf\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.891782 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.891583    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344707b1-2bf5-4481-8522-afe6d996c26c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.891782 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.891688    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.891782 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.891719    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.891782 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.891742    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344707b1-2bf5-4481-8522-afe6d996c26c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.891959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.891847    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.891959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.891876    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344707b1-2bf5-4481-8522-afe6d996c26c-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.891959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.891897    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.993195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.993089    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344707b1-2bf5-4481-8522-afe6d996c26c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.993195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.993145    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.993195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.993160    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344707b1-2bf5-4481-8522-afe6d996c26c-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.993195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.993178    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.993561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.993350    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.993561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.993390    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxmf\" (UniqueName: \"kubernetes.io/projected/344707b1-2bf5-4481-8522-afe6d996c26c-kube-api-access-hhxmf\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.993561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.993439    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344707b1-2bf5-4481-8522-afe6d996c26c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.993561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.993490    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.993561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.993517    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.993825 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.993586    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.993825 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.993716    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.993974 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.993951    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.994046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.993974    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.994244 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.994221    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344707b1-2bf5-4481-8522-afe6d996c26c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.995620 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.995593    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:08.995758 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:08.995739    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344707b1-2bf5-4481-8522-afe6d996c26c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:09.006569 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:09.006546    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344707b1-2bf5-4481-8522-afe6d996c26c-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:09.006703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:09.006597    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhxmf\" (UniqueName: \"kubernetes.io/projected/344707b1-2bf5-4481-8522-afe6d996c26c-kube-api-access-hhxmf\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hl6bc\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:09.057421 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:09.057391    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:09.342570 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:09.342542    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"]
Oct 31 13:23:09.343778 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:23:09.343752    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344707b1_2bf5_4481_8522_afe6d996c26c.slice/crio-f898efaf12aa9e1a006e9239e08e0711cbc617fc8aae342129b0851b982ae9f7 WatchSource:0}: Error finding container f898efaf12aa9e1a006e9239e08e0711cbc617fc8aae342129b0851b982ae9f7: Status 404 returned error can't find the container with id f898efaf12aa9e1a006e9239e08e0711cbc617fc8aae342129b0851b982ae9f7
Oct 31 13:23:09.346081 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:09.346049    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:23:09.346197 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:09.346128    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:23:09.346197 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:09.346171    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:23:10.056069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:10.056030    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc" event={"ID":"344707b1-2bf5-4481-8522-afe6d996c26c","Type":"ContainerStarted","Data":"84460f572da86226085fdde211fd222a3ab7b0fb7d4c7d523533ef909b60cb9b"}
Oct 31 13:23:10.056614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:10.056088    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc" event={"ID":"344707b1-2bf5-4481-8522-afe6d996c26c","Type":"ContainerStarted","Data":"f898efaf12aa9e1a006e9239e08e0711cbc617fc8aae342129b0851b982ae9f7"}
Oct 31 13:23:10.058465 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:10.058447    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:10.059999 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:10.059976    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 13:23:10.060106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:10.060019    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc" podUID="344707b1-2bf5-4481-8522-afe6d996c26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 13:23:10.085381 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:10.085327    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc" podStartSLOduration=2.085286422 podStartE2EDuration="2.085286422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:10.083417393 +0000 UTC m=+1189.702076554" watchObservedRunningTime="2026-10-31 13:23:10.085286422 +0000 UTC m=+1189.703945583"
Oct 31 13:23:11.057854 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:11.057819    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 13:23:11.058248 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:11.057872    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc" podUID="344707b1-2bf5-4481-8522-afe6d996c26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 13:23:12.058072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:12.058038    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 13:23:12.058489 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:12.058091    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc" podUID="344707b1-2bf5-4481-8522-afe6d996c26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 13:23:13.058650 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:13.058612    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 13:23:13.059036 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:13.058669    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc" podUID="344707b1-2bf5-4481-8522-afe6d996c26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 13:23:14.057908 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:14.057872    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 13:23:14.058068 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:14.057925    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc" podUID="344707b1-2bf5-4481-8522-afe6d996c26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 13:23:15.057796 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:15.057754    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 13:23:15.058171 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:15.057839    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc" podUID="344707b1-2bf5-4481-8522-afe6d996c26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 13:23:16.062515 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:16.062482    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:16.062949 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:16.062794    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:16.063497 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:16.063477    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 13:23:20.652373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:20.652344    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:23:20.655733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:20.655712    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:23:20.658740 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:20.658716    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:23:20.662111 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:20.662092    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:23:20.667703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:20.667684    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:23:20.671256 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:20.671236    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:23:20.736012 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:23:20.735971    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:23:23.743957 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.743917    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"]
Oct 31 13:23:23.749230 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.749205    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.761067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.761041    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"]
Oct 31 13:23:23.849620 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.849579    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.849792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.849629    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.849792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.849727    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.849912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.849792    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.849912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.849883    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.850027 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.849919    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.850027 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.849951    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2060533a-c13a-4a2e-bcd0-c904960ed76c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.850125 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.850028    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.850125 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.850092    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvb66\" (UniqueName: \"kubernetes.io/projected/2060533a-c13a-4a2e-bcd0-c904960ed76c-kube-api-access-zvb66\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.950673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.950632    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.950673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.950664    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.950921 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.950714    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.950921 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.950774    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.950921 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.950800    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.950921 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.950823    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.950921 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.950852    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2060533a-c13a-4a2e-bcd0-c904960ed76c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.950921 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.950892    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.951223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.950957    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvb66\" (UniqueName: \"kubernetes.io/projected/2060533a-c13a-4a2e-bcd0-c904960ed76c-kube-api-access-zvb66\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.951223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.951087    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.951223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.951106    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.951411 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.951390    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.951767 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.951743    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.951904 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.951832    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2060533a-c13a-4a2e-bcd0-c904960ed76c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.953560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.953540    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.953732 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.953717    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.966818 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.966792    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvb66\" (UniqueName: \"kubernetes.io/projected/2060533a-c13a-4a2e-bcd0-c904960ed76c-kube-api-access-zvb66\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:23.967341 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:23.967317    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5c26q\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:24.062803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:24.062723    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:24.347928 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:24.347897    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"]
Oct 31 13:23:24.349816 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:23:24.349791    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2060533a_c13a_4a2e_bcd0_c904960ed76c.slice/crio-77b62480f6bac77fc765f3e1bed3a4f35c7ef0c9a5c431ca5635d64734eed5e0 WatchSource:0}: Error finding container 77b62480f6bac77fc765f3e1bed3a4f35c7ef0c9a5c431ca5635d64734eed5e0: Status 404 returned error can't find the container with id 77b62480f6bac77fc765f3e1bed3a4f35c7ef0c9a5c431ca5635d64734eed5e0
Oct 31 13:23:24.352168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:24.352132    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:23:24.352261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:24.352213    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:23:24.352261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:24.352252    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:23:25.155471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:25.155236    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" event={"ID":"2060533a-c13a-4a2e-bcd0-c904960ed76c","Type":"ContainerStarted","Data":"cc7d377c8c85036a57ca2bfe3a90c15a085bf189021f88a04b322a892afce7ec"}
Oct 31 13:23:25.155471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:25.155321    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" event={"ID":"2060533a-c13a-4a2e-bcd0-c904960ed76c","Type":"ContainerStarted","Data":"77b62480f6bac77fc765f3e1bed3a4f35c7ef0c9a5c431ca5635d64734eed5e0"}
Oct 31 13:23:25.184126 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:25.184061    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podStartSLOduration=2.184041527 podStartE2EDuration="2.184041527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:25.179959812 +0000 UTC m=+1204.798618979" watchObservedRunningTime="2026-10-31 13:23:25.184041527 +0000 UTC m=+1204.802700692"
Oct 31 13:23:26.063534 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:26.063494    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:26.065045 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:26.065021    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5c26q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 13:23:26.065159 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:26.065067    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 13:23:27.063277 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:27.063236    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5c26q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 13:23:27.063723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:27.063333    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 13:23:28.064318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:28.063770    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5c26q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 13:23:28.064318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:28.063849    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 13:23:29.063950 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:29.063914    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5c26q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 13:23:29.064232 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:29.064201    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 13:23:30.068378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:30.068264    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5c26q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 13:23:30.068378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:30.068355    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 13:23:32.106576 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:32.106511    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5c26q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:23:32.107312 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:32.107258    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:23:33.108152 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:33.108116    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5c26q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:23:33.108668 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:33.108213    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:23:34.109319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:34.108786    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5c26q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:23:34.109319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:34.108851    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:23:35.164317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:35.156665    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5c26q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:23:35.164317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:35.156737    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:23:36.163280 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:36.161956    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5c26q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:23:36.163280 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:36.162013    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:23:37.207465 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:37.207425    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5c26q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:23:37.208137 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:37.208102    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:23:38.234855 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:38.234816    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5c26q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:23:38.235470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:38.235433    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:23:39.062968 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:39.062928    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:39.276468 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:39.276436    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5c26q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:23:39.276895 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:39.276560    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:23:40.150982 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:40.150949    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:40.162007 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:40.161977    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:23:44.606673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:44.606636    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f57567d7b-4vrrf"]
Oct 31 13:23:44.610522 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:44.610500    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f57567d7b-4vrrf"
Oct 31 13:23:44.620494 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:44.620459    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f57567d7b-4vrrf"]
Oct 31 13:23:44.620637 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:44.620562    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wp529\""
Oct 31 13:23:44.620706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:44.620646    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 13:23:44.696322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:44.696256    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq29q\" (UniqueName: \"kubernetes.io/projected/2d5c423b-4fd2-408b-acce-dfa021e19cb4-kube-api-access-mq29q\") pod \"authorino-5f57567d7b-4vrrf\" (UID: \"2d5c423b-4fd2-408b-acce-dfa021e19cb4\") " pod="kuadrant-system/authorino-5f57567d7b-4vrrf"
Oct 31 13:23:44.696490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:44.696427    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d5c423b-4fd2-408b-acce-dfa021e19cb4-tls-cert\") pod \"authorino-5f57567d7b-4vrrf\" (UID: \"2d5c423b-4fd2-408b-acce-dfa021e19cb4\") " pod="kuadrant-system/authorino-5f57567d7b-4vrrf"
Oct 31 13:23:44.797140 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:44.797099    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq29q\" (UniqueName: \"kubernetes.io/projected/2d5c423b-4fd2-408b-acce-dfa021e19cb4-kube-api-access-mq29q\") pod \"authorino-5f57567d7b-4vrrf\" (UID: \"2d5c423b-4fd2-408b-acce-dfa021e19cb4\") " pod="kuadrant-system/authorino-5f57567d7b-4vrrf"
Oct 31 13:23:44.797337 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:44.797185    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d5c423b-4fd2-408b-acce-dfa021e19cb4-tls-cert\") pod \"authorino-5f57567d7b-4vrrf\" (UID: \"2d5c423b-4fd2-408b-acce-dfa021e19cb4\") " pod="kuadrant-system/authorino-5f57567d7b-4vrrf"
Oct 31 13:23:44.799832 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:44.799804    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d5c423b-4fd2-408b-acce-dfa021e19cb4-tls-cert\") pod \"authorino-5f57567d7b-4vrrf\" (UID: \"2d5c423b-4fd2-408b-acce-dfa021e19cb4\") " pod="kuadrant-system/authorino-5f57567d7b-4vrrf"
Oct 31 13:23:44.810928 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:44.810900    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq29q\" (UniqueName: \"kubernetes.io/projected/2d5c423b-4fd2-408b-acce-dfa021e19cb4-kube-api-access-mq29q\") pod \"authorino-5f57567d7b-4vrrf\" (UID: \"2d5c423b-4fd2-408b-acce-dfa021e19cb4\") " pod="kuadrant-system/authorino-5f57567d7b-4vrrf"
Oct 31 13:23:44.924631 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:44.924596    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f57567d7b-4vrrf"
Oct 31 13:23:45.206445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:45.206413    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f57567d7b-4vrrf"]
Oct 31 13:23:45.407635 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:45.407596    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f57567d7b-4vrrf" event={"ID":"2d5c423b-4fd2-408b-acce-dfa021e19cb4","Type":"ContainerStarted","Data":"2f9957d8d019f3fb43f49876b0b3dbd42f7640e188ecefac46cd27d5b6b28c6f"}
Oct 31 13:23:46.414914 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:46.414817    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f57567d7b-4vrrf" event={"ID":"2d5c423b-4fd2-408b-acce-dfa021e19cb4","Type":"ContainerStarted","Data":"f86478b40a77da828028ef05bb7f60b81e2714469f79252f36ab75937b797ea9"}
Oct 31 13:23:46.435305 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:23:46.435226    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f57567d7b-4vrrf" podStartSLOduration=1.507161575 podStartE2EDuration="2.435213205s" totalImagesPullingTime="928.05163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:46.432943251 +0000 UTC m=+1226.051602428" watchObservedRunningTime="2026-10-31 13:23:46.435213205 +0000 UTC m=+1226.053872423"
Oct 31 13:23:50.736228 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:23:50.736189    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:24:20.731162 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:24:20.731121    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:24:48.200526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:48.200493    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"]
Oct 31 13:24:48.204229 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:48.204204    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"
Oct 31 13:24:48.215625 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:48.215589    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"]
Oct 31 13:24:48.227078 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:48.227051    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-c82140\""
Oct 31 13:24:48.246116 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:48.246094    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-c82140-dockercfg-7xrq5\""
Oct 31 13:24:48.291578 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:48.291544    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84174b11-1866-4f83-ad10-84a6288b53dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw\" (UID: \"84174b11-1866-4f83-ad10-84a6288b53dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"
Oct 31 13:24:48.291578 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:48.291583    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk529\" (UniqueName: \"kubernetes.io/projected/84174b11-1866-4f83-ad10-84a6288b53dd-kube-api-access-qk529\") pod \"payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw\" (UID: \"84174b11-1866-4f83-ad10-84a6288b53dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"
Oct 31 13:24:48.392145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:48.392104    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84174b11-1866-4f83-ad10-84a6288b53dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw\" (UID: \"84174b11-1866-4f83-ad10-84a6288b53dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"
Oct 31 13:24:48.392145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:48.392146    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk529\" (UniqueName: \"kubernetes.io/projected/84174b11-1866-4f83-ad10-84a6288b53dd-kube-api-access-qk529\") pod \"payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw\" (UID: \"84174b11-1866-4f83-ad10-84a6288b53dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"
Oct 31 13:24:48.392920 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:48.392900    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84174b11-1866-4f83-ad10-84a6288b53dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw\" (UID: \"84174b11-1866-4f83-ad10-84a6288b53dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"
Oct 31 13:24:48.407513 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:48.407490    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk529\" (UniqueName: \"kubernetes.io/projected/84174b11-1866-4f83-ad10-84a6288b53dd-kube-api-access-qk529\") pod \"payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw\" (UID: \"84174b11-1866-4f83-ad10-84a6288b53dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"
Oct 31 13:24:48.520178 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:48.520091    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"
Oct 31 13:24:48.804231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:48.803381    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"]
Oct 31 13:24:48.806340 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:24:48.806282    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84174b11_1866_4f83_ad10_84a6288b53dd.slice/crio-734c9990224c9a73d9bd7425284b2584b68b79a14696413791660bc396480876 WatchSource:0}: Error finding container 734c9990224c9a73d9bd7425284b2584b68b79a14696413791660bc396480876: Status 404 returned error can't find the container with id 734c9990224c9a73d9bd7425284b2584b68b79a14696413791660bc396480876
Oct 31 13:24:49.336366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:49.336330    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"]
Oct 31 13:24:49.342461 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:49.341423    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:49.389417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:49.389387    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"]
Oct 31 13:24:49.445873 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:49.445841    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-6784d6-serving-cert\""
Oct 31 13:24:49.511970 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:49.508111    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5ht4\" (UniqueName: \"kubernetes.io/projected/04449154-f556-4576-b8c1-1a4ec39c960a-kube-api-access-x5ht4\") pod \"maas-api-e2e-worker-w5-6784d6-675775b557-v99vq\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:49.511970 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:49.508166    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-maas-api-tls\") pod \"maas-api-e2e-worker-w5-6784d6-675775b557-v99vq\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:49.511970 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:49.508383    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04449154-f556-4576-b8c1-1a4ec39c960a-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-6784d6-675775b557-v99vq\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:49.511970 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:49.508427    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-metrics-certs\") pod \"maas-api-e2e-worker-w5-6784d6-675775b557-v99vq\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:49.609577 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:49.609487    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5ht4\" (UniqueName: \"kubernetes.io/projected/04449154-f556-4576-b8c1-1a4ec39c960a-kube-api-access-x5ht4\") pod \"maas-api-e2e-worker-w5-6784d6-675775b557-v99vq\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:49.609577 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:49.609547    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-maas-api-tls\") pod \"maas-api-e2e-worker-w5-6784d6-675775b557-v99vq\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:49.609803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:49.609604    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04449154-f556-4576-b8c1-1a4ec39c960a-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-6784d6-675775b557-v99vq\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:49.609803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:49.609655    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-metrics-certs\") pod \"maas-api-e2e-worker-w5-6784d6-675775b557-v99vq\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:49.609803 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:24:49.609739    4660 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-serving-cert: secret "maas-api-e2e-worker-w5-6784d6-serving-cert" not found
Oct 31 13:24:49.609916 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:24:49.609809    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-maas-api-tls podName:04449154-f556-4576-b8c1-1a4ec39c960a nodeName:}" failed. No retries permitted until 2026-10-31 13:24:50.109788746 +0000 UTC m=+1289.728447894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-maas-api-tls") pod "maas-api-e2e-worker-w5-6784d6-675775b557-v99vq" (UID: "04449154-f556-4576-b8c1-1a4ec39c960a") : secret "maas-api-e2e-worker-w5-6784d6-serving-cert" not found
Oct 31 13:24:49.610496 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:49.610469    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04449154-f556-4576-b8c1-1a4ec39c960a-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-6784d6-675775b557-v99vq\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:49.617817 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:49.617772    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-metrics-certs\") pod \"maas-api-e2e-worker-w5-6784d6-675775b557-v99vq\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:49.623703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:49.623682    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5ht4\" (UniqueName: \"kubernetes.io/projected/04449154-f556-4576-b8c1-1a4ec39c960a-kube-api-access-x5ht4\") pod \"maas-api-e2e-worker-w5-6784d6-675775b557-v99vq\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:49.797492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:49.797409    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw" event={"ID":"84174b11-1866-4f83-ad10-84a6288b53dd","Type":"ContainerStarted","Data":"734c9990224c9a73d9bd7425284b2584b68b79a14696413791660bc396480876"}
Oct 31 13:24:50.122499 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.122359    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-maas-api-tls\") pod \"maas-api-e2e-worker-w5-6784d6-675775b557-v99vq\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:50.129330 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.125392    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-maas-api-tls\") pod \"maas-api-e2e-worker-w5-6784d6-675775b557-v99vq\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:50.282528 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.282491    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:50.582654 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.582624    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"]
Oct 31 13:24:50.585275 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:24:50.585239    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04449154_f556_4576_b8c1_1a4ec39c960a.slice/crio-068a1d8dc0a96df695be0f1385f32db6dfecf2d306f675564bb5eb1022656124 WatchSource:0}: Error finding container 068a1d8dc0a96df695be0f1385f32db6dfecf2d306f675564bb5eb1022656124: Status 404 returned error can't find the container with id 068a1d8dc0a96df695be0f1385f32db6dfecf2d306f675564bb5eb1022656124
Oct 31 13:24:50.681987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.679948    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"]
Oct 31 13:24:50.689463 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.689425    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 13:24:50.695130 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.694930    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"]
Oct 31 13:24:50.697448 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.697409    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-c82140-serving-cert\""
Oct 31 13:24:50.744503 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:24:50.744462    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:24:50.809436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.809401    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq" event={"ID":"04449154-f556-4576-b8c1-1a4ec39c960a","Type":"ContainerStarted","Data":"068a1d8dc0a96df695be0f1385f32db6dfecf2d306f675564bb5eb1022656124"}
Oct 31 13:24:50.835026 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.834945    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgjqh\" (UniqueName: \"kubernetes.io/projected/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-kube-api-access-jgjqh\") pod \"maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz\" (UID: \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 13:24:50.835178 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.835106    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-metrics-certs\") pod \"maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz\" (UID: \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 13:24:50.835277 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.835252    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz\" (UID: \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 13:24:50.835409 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.835364    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz\" (UID: \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 13:24:50.946822 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.946776    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz\" (UID: \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 13:24:50.947128 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.946876    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgjqh\" (UniqueName: \"kubernetes.io/projected/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-kube-api-access-jgjqh\") pod \"maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz\" (UID: \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 13:24:50.947128 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.946940    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-metrics-certs\") pod \"maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz\" (UID: \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 13:24:50.947128 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.947012    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz\" (UID: \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 13:24:50.948410 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.948380    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz\" (UID: \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 13:24:50.962656 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.962617    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz\" (UID: \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 13:24:50.962815 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.962740    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-metrics-certs\") pod \"maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz\" (UID: \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 13:24:50.964148 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:50.964094    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgjqh\" (UniqueName: \"kubernetes.io/projected/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-kube-api-access-jgjqh\") pod \"maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz\" (UID: \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 13:24:51.011350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:51.011319    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 13:24:51.318585 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:51.318544    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"]
Oct 31 13:24:51.818663 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:51.818624    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz" event={"ID":"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e","Type":"ContainerStarted","Data":"7972cc00e6429124f86078fcf411100eed836c8fc6640b764ceb8a7961f31a74"}
Oct 31 13:24:51.820374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:51.820345    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq" event={"ID":"04449154-f556-4576-b8c1-1a4ec39c960a","Type":"ContainerStarted","Data":"00cc6457f30436dfd385a27d6e736a3c5bfbe88bbf492ea47d01555da8276611"}
Oct 31 13:24:51.820481 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:51.820420    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:51.821991 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:51.821967    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw" event={"ID":"84174b11-1866-4f83-ad10-84a6288b53dd","Type":"ContainerStarted","Data":"9d5ed341b35fee7a96a7b4bb55cab9e6e67ddc86fcd98e941140e6a8690ae92e"}
Oct 31 13:24:51.822096 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:51.822022    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"
Oct 31 13:24:51.843066 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:51.843008    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq" podStartSLOduration=2.341401893 podStartE2EDuration="2.842991798s" totalImagesPullingTime="501.589905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:51.841484265 +0000 UTC m=+1291.460143431" watchObservedRunningTime="2026-10-31 13:24:51.842991798 +0000 UTC m=+1291.461650966"
Oct 31 13:24:51.863309 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:51.863247    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw" podStartSLOduration=1.230990482 podStartE2EDuration="3.863234151s" totalImagesPullingTime="2.632243669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:51.861384494 +0000 UTC m=+1291.480043672" watchObservedRunningTime="2026-10-31 13:24:51.863234151 +0000 UTC m=+1291.481893316"
Oct 31 13:24:51.899154 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:51.899115    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"]
Oct 31 13:24:51.903074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:51.903047    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 13:24:51.915129 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:51.915103    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-aabeef-serving-cert\""
Oct 31 13:24:51.918937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:51.918911    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"]
Oct 31 13:24:52.061674 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.061586    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e133f4e-385e-46b4-ab50-db03976321e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w\" (UID: \"8e133f4e-385e-46b4-ab50-db03976321e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 13:24:52.061824 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.061690    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e133f4e-385e-46b4-ab50-db03976321e0-maas-api-tls\") pod \"maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w\" (UID: \"8e133f4e-385e-46b4-ab50-db03976321e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 13:24:52.061869 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.061820    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e133f4e-385e-46b4-ab50-db03976321e0-metrics-certs\") pod \"maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w\" (UID: \"8e133f4e-385e-46b4-ab50-db03976321e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 13:24:52.061922 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.061867    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmbqd\" (UniqueName: \"kubernetes.io/projected/8e133f4e-385e-46b4-ab50-db03976321e0-kube-api-access-wmbqd\") pod \"maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w\" (UID: \"8e133f4e-385e-46b4-ab50-db03976321e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 13:24:52.163026 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.162985    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e133f4e-385e-46b4-ab50-db03976321e0-metrics-certs\") pod \"maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w\" (UID: \"8e133f4e-385e-46b4-ab50-db03976321e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 13:24:52.163195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.163037    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmbqd\" (UniqueName: \"kubernetes.io/projected/8e133f4e-385e-46b4-ab50-db03976321e0-kube-api-access-wmbqd\") pod \"maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w\" (UID: \"8e133f4e-385e-46b4-ab50-db03976321e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 13:24:52.163195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.163068    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e133f4e-385e-46b4-ab50-db03976321e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w\" (UID: \"8e133f4e-385e-46b4-ab50-db03976321e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 13:24:52.163195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.163089    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e133f4e-385e-46b4-ab50-db03976321e0-maas-api-tls\") pod \"maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w\" (UID: \"8e133f4e-385e-46b4-ab50-db03976321e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 13:24:52.163886 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.163867    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e133f4e-385e-46b4-ab50-db03976321e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w\" (UID: \"8e133f4e-385e-46b4-ab50-db03976321e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 13:24:52.165522 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.165498    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e133f4e-385e-46b4-ab50-db03976321e0-metrics-certs\") pod \"maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w\" (UID: \"8e133f4e-385e-46b4-ab50-db03976321e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 13:24:52.165656 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.165635    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e133f4e-385e-46b4-ab50-db03976321e0-maas-api-tls\") pod \"maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w\" (UID: \"8e133f4e-385e-46b4-ab50-db03976321e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 13:24:52.176212 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.176189    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmbqd\" (UniqueName: \"kubernetes.io/projected/8e133f4e-385e-46b4-ab50-db03976321e0-kube-api-access-wmbqd\") pod \"maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w\" (UID: \"8e133f4e-385e-46b4-ab50-db03976321e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 13:24:52.218408 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.218374    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 13:24:52.518357 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.518327    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"]
Oct 31 13:24:52.841618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.841528    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w" event={"ID":"8e133f4e-385e-46b4-ab50-db03976321e0","Type":"ContainerStarted","Data":"66d5a7d0c44c144af7398f9a47565ff5bea096754b09d9d8388d1db66651d7a2"}
Oct 31 13:24:52.843550 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.843515    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz" event={"ID":"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e","Type":"ContainerStarted","Data":"5368d6c18711b7d9e12226daa599ccd1bf81b1072e6e83897f2abd82929719af"}
Oct 31 13:24:52.843986 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.843960    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 13:24:52.872413 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:52.872360    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz" podStartSLOduration=2.442589993 podStartE2EDuration="2.872342053s" totalImagesPullingTime="429.75206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:52.869420161 +0000 UTC m=+1292.488079327" watchObservedRunningTime="2026-10-31 13:24:52.872342053 +0000 UTC m=+1292.491001217"
Oct 31 13:24:53.197450 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.197412    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"]
Oct 31 13:24:53.201889 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.201861    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:24:53.209545 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.209461    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-5286ea-serving-cert\""
Oct 31 13:24:53.215628 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.215604    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"]
Oct 31 13:24:53.375143 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.375096    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90939bcc-e6bc-42f3-b535-5d10ec96b1be-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv\" (UID: \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:24:53.375352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.375159    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90939bcc-e6bc-42f3-b535-5d10ec96b1be-maas-api-tls\") pod \"maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv\" (UID: \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:24:53.375352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.375318    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zps47\" (UniqueName: \"kubernetes.io/projected/90939bcc-e6bc-42f3-b535-5d10ec96b1be-kube-api-access-zps47\") pod \"maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv\" (UID: \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:24:53.375448 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.375384    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90939bcc-e6bc-42f3-b535-5d10ec96b1be-metrics-certs\") pod \"maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv\" (UID: \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:24:53.476717 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.476638    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zps47\" (UniqueName: \"kubernetes.io/projected/90939bcc-e6bc-42f3-b535-5d10ec96b1be-kube-api-access-zps47\") pod \"maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv\" (UID: \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:24:53.476717 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.476680    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90939bcc-e6bc-42f3-b535-5d10ec96b1be-metrics-certs\") pod \"maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv\" (UID: \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:24:53.476894 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.476740    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90939bcc-e6bc-42f3-b535-5d10ec96b1be-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv\" (UID: \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:24:53.476894 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.476772    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90939bcc-e6bc-42f3-b535-5d10ec96b1be-maas-api-tls\") pod \"maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv\" (UID: \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:24:53.477439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.477408    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90939bcc-e6bc-42f3-b535-5d10ec96b1be-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv\" (UID: \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:24:53.479252 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.479230    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90939bcc-e6bc-42f3-b535-5d10ec96b1be-metrics-certs\") pod \"maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv\" (UID: \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:24:53.479370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.479247    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90939bcc-e6bc-42f3-b535-5d10ec96b1be-maas-api-tls\") pod \"maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv\" (UID: \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:24:53.489823 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.489801    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zps47\" (UniqueName: \"kubernetes.io/projected/90939bcc-e6bc-42f3-b535-5d10ec96b1be-kube-api-access-zps47\") pod \"maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv\" (UID: \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:24:53.522084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.522056    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:24:53.808184 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.808151    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"]
Oct 31 13:24:53.809021 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:24:53.808996    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90939bcc_e6bc_42f3_b535_5d10ec96b1be.slice/crio-82ffc7fa2e80bedd86e709c78f25ed6f840984f9488f65b67578db8f667bb46d WatchSource:0}: Error finding container 82ffc7fa2e80bedd86e709c78f25ed6f840984f9488f65b67578db8f667bb46d: Status 404 returned error can't find the container with id 82ffc7fa2e80bedd86e709c78f25ed6f840984f9488f65b67578db8f667bb46d
Oct 31 13:24:53.850895 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.850838    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv" event={"ID":"90939bcc-e6bc-42f3-b535-5d10ec96b1be","Type":"ContainerStarted","Data":"82ffc7fa2e80bedd86e709c78f25ed6f840984f9488f65b67578db8f667bb46d"}
Oct 31 13:24:53.852400 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.852376    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w" event={"ID":"8e133f4e-385e-46b4-ab50-db03976321e0","Type":"ContainerStarted","Data":"25675e9bc9c985c18c574b46ac180881ef881bdfdcea5c3f02885cfbc95c4d66"}
Oct 31 13:24:53.852652 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.852632    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 13:24:53.875226 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:53.875181    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w" podStartSLOduration=2.433921282 podStartE2EDuration="2.875169511s" totalImagesPullingTime="441.248229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:53.872802804 +0000 UTC m=+1293.491461968" watchObservedRunningTime="2026-10-31 13:24:53.875169511 +0000 UTC m=+1293.493828675"
Oct 31 13:24:54.860382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:54.860341    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv" event={"ID":"90939bcc-e6bc-42f3-b535-5d10ec96b1be","Type":"ContainerStarted","Data":"baadc5c8f41f0c200fd7a14bb243f3c41d78b05e852225fec0eb3e3abe54c613"}
Oct 31 13:24:54.860858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:54.860537    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:24:54.884500 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:54.884451    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv" podStartSLOduration=1.5005018570000002 podStartE2EDuration="1.884437444s" totalImagesPullingTime="383.935587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:54.880890671 +0000 UTC m=+1294.499549837" watchObservedRunningTime="2026-10-31 13:24:54.884437444 +0000 UTC m=+1294.503096608"
Oct 31 13:24:55.881060 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:55.881029    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"]
Oct 31 13:24:55.885133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:55.885106    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:24:55.893900 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:55.893874    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-7b3ddd-serving-cert\""
Oct 31 13:24:55.897657 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:55.897632    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"]
Oct 31 13:24:56.003703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:56.003652    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99r9d\" (UniqueName: \"kubernetes.io/projected/e425a959-4078-4fdc-88d2-8e6906eef5ab-kube-api-access-99r9d\") pod \"maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2\" (UID: \"e425a959-4078-4fdc-88d2-8e6906eef5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:24:56.003865 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:56.003829    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e425a959-4078-4fdc-88d2-8e6906eef5ab-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2\" (UID: \"e425a959-4078-4fdc-88d2-8e6906eef5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:24:56.004056 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:56.004029    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e425a959-4078-4fdc-88d2-8e6906eef5ab-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2\" (UID: \"e425a959-4078-4fdc-88d2-8e6906eef5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:24:56.004156 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:56.004077    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e425a959-4078-4fdc-88d2-8e6906eef5ab-metrics-certs\") pod \"maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2\" (UID: \"e425a959-4078-4fdc-88d2-8e6906eef5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:24:56.105231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:56.105196    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e425a959-4078-4fdc-88d2-8e6906eef5ab-metrics-certs\") pod \"maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2\" (UID: \"e425a959-4078-4fdc-88d2-8e6906eef5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:24:56.105445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:56.105254    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99r9d\" (UniqueName: \"kubernetes.io/projected/e425a959-4078-4fdc-88d2-8e6906eef5ab-kube-api-access-99r9d\") pod \"maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2\" (UID: \"e425a959-4078-4fdc-88d2-8e6906eef5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:24:56.105445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:56.105305    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e425a959-4078-4fdc-88d2-8e6906eef5ab-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2\" (UID: \"e425a959-4078-4fdc-88d2-8e6906eef5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:24:56.105445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:56.105405    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e425a959-4078-4fdc-88d2-8e6906eef5ab-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2\" (UID: \"e425a959-4078-4fdc-88d2-8e6906eef5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:24:56.106122 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:56.106084    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e425a959-4078-4fdc-88d2-8e6906eef5ab-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2\" (UID: \"e425a959-4078-4fdc-88d2-8e6906eef5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:24:56.107866 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:56.107843    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e425a959-4078-4fdc-88d2-8e6906eef5ab-metrics-certs\") pod \"maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2\" (UID: \"e425a959-4078-4fdc-88d2-8e6906eef5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:24:56.108223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:56.108202    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e425a959-4078-4fdc-88d2-8e6906eef5ab-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2\" (UID: \"e425a959-4078-4fdc-88d2-8e6906eef5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:24:56.123514 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:56.123487    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99r9d\" (UniqueName: \"kubernetes.io/projected/e425a959-4078-4fdc-88d2-8e6906eef5ab-kube-api-access-99r9d\") pod \"maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2\" (UID: \"e425a959-4078-4fdc-88d2-8e6906eef5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:24:56.201447 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:56.201348    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:24:56.492289 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:56.492261    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"]
Oct 31 13:24:56.494888 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:24:56.494856    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode425a959_4078_4fdc_88d2_8e6906eef5ab.slice/crio-7845f0ab8df0a4c0d8d4e0c6415e5900cca65d0bf258e2126e2443f855bd6547 WatchSource:0}: Error finding container 7845f0ab8df0a4c0d8d4e0c6415e5900cca65d0bf258e2126e2443f855bd6547: Status 404 returned error can't find the container with id 7845f0ab8df0a4c0d8d4e0c6415e5900cca65d0bf258e2126e2443f855bd6547
Oct 31 13:24:56.873071 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:56.873035    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2" event={"ID":"e425a959-4078-4fdc-88d2-8e6906eef5ab","Type":"ContainerStarted","Data":"7845f0ab8df0a4c0d8d4e0c6415e5900cca65d0bf258e2126e2443f855bd6547"}
Oct 31 13:24:57.849328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:57.849286    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 13:24:57.882015 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:57.881977    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2" event={"ID":"e425a959-4078-4fdc-88d2-8e6906eef5ab","Type":"ContainerStarted","Data":"a546919faa29bdb48aae76686113c5a2431de0b7db3708df9d115c20bce5e05f"}
Oct 31 13:24:57.882271 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:57.882246    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:24:58.858448 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:58.858419    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 13:24:58.880231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:58.880176    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2" podStartSLOduration=3.355580716 podStartE2EDuration="3.880156753s" totalImagesPullingTime="524.576037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:57.908490103 +0000 UTC m=+1297.527149267" watchObservedRunningTime="2026-10-31 13:24:58.880156753 +0000 UTC m=+1298.498815921"
Oct 31 13:24:59.867358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:24:59.867326    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 13:25:00.873564 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:00.873536    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:25:02.855535 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:02.855500    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"
Oct 31 13:25:03.893693 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:03.893664    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:25:20.737498 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:25:20.737460    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:25:27.648346 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.648312    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"]
Oct 31 13:25:27.655878 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.655855    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.668908 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.668883    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8b2f0a\"/\"kube-root-ca.crt\""
Oct 31 13:25:27.668908 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.668901    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8b2f0a\"/\"default-dockercfg-mxnz2\""
Oct 31 13:25:27.668908 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.668902    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8b2f0a\"/\"openshift-service-ca.crt\""
Oct 31 13:25:27.669171 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.668960    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8b2f0a\"/\"simulator-w4-8b2f0a-kserve-self-signed-certs\""
Oct 31 13:25:27.672258 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.672237    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"]
Oct 31 13:25:27.767947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.767913    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd80067-027a-40ab-be2f-63e7597a72f3-tls-certs\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.767947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.767960    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-model-cache\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.768175 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.768117    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g96jv\" (UniqueName: \"kubernetes.io/projected/0fd80067-027a-40ab-be2f-63e7597a72f3-kube-api-access-g96jv\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.768244 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.768225    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-tmp-dir\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.768284 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.768261    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-dshm\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.768351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.768317    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-home\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.869731 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.869696    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd80067-027a-40ab-be2f-63e7597a72f3-tls-certs\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.869938 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.869748    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-model-cache\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.869938 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.869800    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g96jv\" (UniqueName: \"kubernetes.io/projected/0fd80067-027a-40ab-be2f-63e7597a72f3-kube-api-access-g96jv\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.869938 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.869901    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-tmp-dir\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.869938 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.869925    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-dshm\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.870144 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.869966    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-home\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.870235 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.870208    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-model-cache\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.870322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.870261    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-tmp-dir\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.870466 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.870447    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-home\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.872644 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.872611    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd80067-027a-40ab-be2f-63e7597a72f3-tls-certs\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.873706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.873680    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-dshm\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.885446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.885421    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g96jv\" (UniqueName: \"kubernetes.io/projected/0fd80067-027a-40ab-be2f-63e7597a72f3-kube-api-access-g96jv\") pod \"simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:27.968471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:27.968391    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:28.252269 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:28.252236    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"]
Oct 31 13:25:28.253095 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:25:28.253065    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fd80067_027a_40ab_be2f_63e7597a72f3.slice/crio-278fdb8993d9d39c6d8393526550a9f931f7bf6139a919433d4493264decb83c WatchSource:0}: Error finding container 278fdb8993d9d39c6d8393526550a9f931f7bf6139a919433d4493264decb83c: Status 404 returned error can't find the container with id 278fdb8993d9d39c6d8393526550a9f931f7bf6139a919433d4493264decb83c
Oct 31 13:25:29.130757 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:29.130716    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" event={"ID":"0fd80067-027a-40ab-be2f-63e7597a72f3","Type":"ContainerStarted","Data":"278fdb8993d9d39c6d8393526550a9f931f7bf6139a919433d4493264decb83c"}
Oct 31 13:25:30.140314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:30.140267    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" event={"ID":"0fd80067-027a-40ab-be2f-63e7597a72f3","Type":"ContainerStarted","Data":"526fd02de6133614fba2e1aeb172daec9d81149c56e2cff8bacb5ea026e624e1"}
Oct 31 13:25:30.162091 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:30.162028    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podStartSLOduration=1.486071584 podStartE2EDuration="3.162010651s" totalImagesPullingTime="1.675939067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:30.160468367 +0000 UTC m=+1329.779127531" watchObservedRunningTime="2026-10-31 13:25:30.162010651 +0000 UTC m=+1329.780669818"
Oct 31 13:25:30.968961 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:30.968924    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:37.969375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:37.969341    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:37.983976 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:37.983949    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:38.207599 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:38.207563    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:25:50.737442 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:25:50.737401    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:25:52.158108 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.158073    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"]
Oct 31 13:25:52.162538 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.162515    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.168046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.168026    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7b3ddd\"/\"kube-root-ca.crt\""
Oct 31 13:25:52.168207 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.168050    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7b3ddd\"/\"default-dockercfg-bqk6t\""
Oct 31 13:25:52.168207 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.168086    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7b3ddd\"/\"openshift-service-ca.crt\""
Oct 31 13:25:52.168207 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.168087    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7b3ddd\"/\"premium-w0-7b3ddd-kserve-self-signed-certs\""
Oct 31 13:25:52.179690 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.179666    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"]
Oct 31 13:25:52.265882 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.265847    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-model-cache\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.266071 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.265893    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-tmp-dir\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.266071 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.265923    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-home\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.266195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.266101    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/150de627-a508-4a67-8be3-194452c9325d-tls-certs\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.266195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.266164    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxlxr\" (UniqueName: \"kubernetes.io/projected/150de627-a508-4a67-8be3-194452c9325d-kube-api-access-jxlxr\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.266327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.266223    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-dshm\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.367146 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.367106    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-dshm\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.367401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.367182    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-model-cache\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.367401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.367209    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-tmp-dir\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.367401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.367233    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-home\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.367401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.367389    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/150de627-a508-4a67-8be3-194452c9325d-tls-certs\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.367735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.367707    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-home\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.367735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.367727    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-model-cache\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.367853 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.367779    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-tmp-dir\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.367910 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.367874    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxlxr\" (UniqueName: \"kubernetes.io/projected/150de627-a508-4a67-8be3-194452c9325d-kube-api-access-jxlxr\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.373331 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.370009    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-dshm\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.373331 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.370083    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/150de627-a508-4a67-8be3-194452c9325d-tls-certs\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.380976 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.380950    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxlxr\" (UniqueName: \"kubernetes.io/projected/150de627-a508-4a67-8be3-194452c9325d-kube-api-access-jxlxr\") pod \"premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") " pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.477344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.477251    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:52.774504 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:52.774478    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"]
Oct 31 13:25:52.776348 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:25:52.776321    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod150de627_a508_4a67_8be3_194452c9325d.slice/crio-d78ac0abc2cf9d72d1c3e9a619a15e362030026d9ff6dfcf701c8806e497c2c7 WatchSource:0}: Error finding container d78ac0abc2cf9d72d1c3e9a619a15e362030026d9ff6dfcf701c8806e497c2c7: Status 404 returned error can't find the container with id d78ac0abc2cf9d72d1c3e9a619a15e362030026d9ff6dfcf701c8806e497c2c7
Oct 31 13:25:53.358402 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:53.358356    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" event={"ID":"150de627-a508-4a67-8be3-194452c9325d","Type":"ContainerStarted","Data":"83f935155f328e84a42ed360654630e1c78eca6445bfb9b1266bc9795ef3b02f"}
Oct 31 13:25:53.358878 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:53.358410    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" event={"ID":"150de627-a508-4a67-8be3-194452c9325d","Type":"ContainerStarted","Data":"d78ac0abc2cf9d72d1c3e9a619a15e362030026d9ff6dfcf701c8806e497c2c7"}
Oct 31 13:25:53.385980 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:53.385915    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podStartSLOduration=1.385893393 podStartE2EDuration="1.385893393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:53.383203032 +0000 UTC m=+1353.001862198" watchObservedRunningTime="2026-10-31 13:25:53.385893393 +0000 UTC m=+1353.004552558"
Oct 31 13:25:53.477769 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:53.477371    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:25:57.757656 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:57.757619    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"]
Oct 31 13:25:57.761697 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:57.761679    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:57.772036 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:57.771992    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8b2f0a\"/\"premium-w4-8b2f0a-kserve-self-signed-certs\""
Oct 31 13:25:57.775545 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:57.775521    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"]
Oct 31 13:25:57.924662 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:57.924627    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-model-cache\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:57.924662 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:57.924662    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce80f7f-51e8-46f3-add6-37bbb3a0adba-tls-certs\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:57.924913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:57.924682    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-home\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:57.924913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:57.924832    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-tmp-dir\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:57.924913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:57.924876    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt4qm\" (UniqueName: \"kubernetes.io/projected/bce80f7f-51e8-46f3-add6-37bbb3a0adba-kube-api-access-zt4qm\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:57.925088 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:57.924976    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-dshm\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:58.026088 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:58.026018    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-dshm\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:58.026088 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:58.026083    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-model-cache\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:58.026340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:58.026101    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce80f7f-51e8-46f3-add6-37bbb3a0adba-tls-certs\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:58.026340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:58.026116    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-home\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:58.026340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:58.026147    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-tmp-dir\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:58.026340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:58.026164    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt4qm\" (UniqueName: \"kubernetes.io/projected/bce80f7f-51e8-46f3-add6-37bbb3a0adba-kube-api-access-zt4qm\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:58.026557 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:58.026520    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-model-cache\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:58.026612 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:58.026582    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-home\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:58.026683 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:58.026660    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-tmp-dir\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:58.028414 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:58.028393    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-dshm\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:58.028819 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:58.028796    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce80f7f-51e8-46f3-add6-37bbb3a0adba-tls-certs\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:58.039798 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:58.039772    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt4qm\" (UniqueName: \"kubernetes.io/projected/bce80f7f-51e8-46f3-add6-37bbb3a0adba-kube-api-access-zt4qm\") pod \"premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") " pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:58.077618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:58.077592    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:25:58.389080 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:58.389051    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"]
Oct 31 13:25:58.390555 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:25:58.390525    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbce80f7f_51e8_46f3_add6_37bbb3a0adba.slice/crio-a18aec24480098a5ca4afac7f547eb16ae9a82bdd452e11aa74ab5192e34b25a WatchSource:0}: Error finding container a18aec24480098a5ca4afac7f547eb16ae9a82bdd452e11aa74ab5192e34b25a: Status 404 returned error can't find the container with id a18aec24480098a5ca4afac7f547eb16ae9a82bdd452e11aa74ab5192e34b25a
Oct 31 13:25:59.403360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:59.403309    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" event={"ID":"bce80f7f-51e8-46f3-add6-37bbb3a0adba","Type":"ContainerStarted","Data":"e4a945cfaa7729b72fba229fb6cd54809980904cff7b87ad50237f5d6ba5a09c"}
Oct 31 13:25:59.403360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:59.403366    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" event={"ID":"bce80f7f-51e8-46f3-add6-37bbb3a0adba","Type":"ContainerStarted","Data":"a18aec24480098a5ca4afac7f547eb16ae9a82bdd452e11aa74ab5192e34b25a"}
Oct 31 13:25:59.430775 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:25:59.430723    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podStartSLOduration=2.430709937 podStartE2EDuration="2.430709937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:59.428047845 +0000 UTC m=+1359.046707017" watchObservedRunningTime="2026-10-31 13:25:59.430709937 +0000 UTC m=+1359.049369100"
Oct 31 13:26:00.078423 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:00.078389    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:26:02.477965 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:02.477920    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:26:02.490876 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:02.490849    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:26:03.446565 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:03.446533    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:26:08.078037 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:08.077997    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:26:08.091565 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:08.091540    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:26:08.493709 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:08.493676    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:26:13.333322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.333273    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"]
Oct 31 13:26:13.337267 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.337244    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.342135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.342102    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-c82140-openshift-default-dockercfg-bmc79\""
Oct 31 13:26:13.351985 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.351959    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"]
Oct 31 13:26:13.386388 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.386351    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.386559 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.386408    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.386559 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.386520    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.386559 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.386546    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7clb\" (UniqueName: \"kubernetes.io/projected/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-kube-api-access-b7clb\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.386728 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.386649    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.386728 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.386700    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.386814 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.386802    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.386856 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.386830    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.386895 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.386883    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.487917 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.487880    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.487917 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.487918    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7clb\" (UniqueName: \"kubernetes.io/projected/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-kube-api-access-b7clb\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.488167 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.487946    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.488167 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.487968    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.488167 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.488009    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.488167 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.488030    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.488167 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.488052    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.488167 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.488108    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.488167 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.488161    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.488679 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.488650    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.488778 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.488692    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.488778 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.488730    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.488896 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.488853    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.488896 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.488856    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.490359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.490332    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.490737 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.490716    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.501889 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.501852    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.502021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.501987    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7clb\" (UniqueName: \"kubernetes.io/projected/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-kube-api-access-b7clb\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.652935 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.652904    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:13.943076 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.943049    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"]
Oct 31 13:26:13.946528 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:26:13.946493    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3e7468_c108_44b5_8fae_a38ddc3d2e42.slice/crio-8d1392fb76d3e55cd560d1b90a3b32f64e6b3d6834cae835a848c7b6d8bc94dc WatchSource:0}: Error finding container 8d1392fb76d3e55cd560d1b90a3b32f64e6b3d6834cae835a848c7b6d8bc94dc: Status 404 returned error can't find the container with id 8d1392fb76d3e55cd560d1b90a3b32f64e6b3d6834cae835a848c7b6d8bc94dc
Oct 31 13:26:13.949177 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.949140    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:26:13.949328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.949227    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:26:13.949328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:13.949275    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:26:14.559230 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:14.559190    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8" event={"ID":"9d3e7468-c108-44b5-8fae-a38ddc3d2e42","Type":"ContainerStarted","Data":"7c687929ea3cc33327ed487de305acb450c50d8bbd0adbd6986fe62d59abe612"}
Oct 31 13:26:14.559641 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:14.559255    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8" event={"ID":"9d3e7468-c108-44b5-8fae-a38ddc3d2e42","Type":"ContainerStarted","Data":"8d1392fb76d3e55cd560d1b90a3b32f64e6b3d6834cae835a848c7b6d8bc94dc"}
Oct 31 13:26:14.585971 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:14.585917    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8" podStartSLOduration=1.585904498 podStartE2EDuration="1.585904498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:14.583896109 +0000 UTC m=+1374.202555273" watchObservedRunningTime="2026-10-31 13:26:14.585904498 +0000 UTC m=+1374.204563662"
Oct 31 13:26:14.653749 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:14.653694    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:15.655162 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:15.655125    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 13:26:15.655558 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:15.655185    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8" podUID="9d3e7468-c108-44b5-8fae-a38ddc3d2e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 13:26:16.653821 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:16.653790    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 13:26:16.654003 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:16.653840    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8" podUID="9d3e7468-c108-44b5-8fae-a38ddc3d2e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 13:26:17.653952 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:17.653914    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 13:26:17.654444 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:17.653969    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8" podUID="9d3e7468-c108-44b5-8fae-a38ddc3d2e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 13:26:18.653634 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:18.653597    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 13:26:18.653822 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:18.653660    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8" podUID="9d3e7468-c108-44b5-8fae-a38ddc3d2e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 13:26:19.653753 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:19.653706    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 13:26:19.654398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:19.654361    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8" podUID="9d3e7468-c108-44b5-8fae-a38ddc3d2e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 13:26:20.744382 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:26:20.744339    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:26:20.801521 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:20.801489    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:20.801965 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:20.801937    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:20.803206 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:26:20.803186    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:26:50.733464 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:26:50.733418    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:27:08.793696 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:08.793661    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"]
Oct 31 13:27:08.797782 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:08.797760    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:08.808467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:08.808446    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-aabeef\"/\"kube-root-ca.crt\""
Oct 31 13:27:08.808652 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:08.808543    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aabeef\"/\"distinct-w6-aabeef-kserve-self-signed-certs\""
Oct 31 13:27:08.808652 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:08.808611    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-aabeef\"/\"openshift-service-ca.crt\""
Oct 31 13:27:08.808797 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:08.808679    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aabeef\"/\"default-dockercfg-9t4bh\""
Oct 31 13:27:08.809213 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:08.809187    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"]
Oct 31 13:27:08.899429 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:08.899398    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-dshm\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:08.899597 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:08.899461    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-model-cache\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:08.899597 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:08.899521    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnw2n\" (UniqueName: \"kubernetes.io/projected/38522f51-cc0e-4026-a6f3-a9be0b429561-kube-api-access-bnw2n\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:08.899682 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:08.899617    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-home\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:08.899682 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:08.899665    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-tmp-dir\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:08.899758 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:08.899744    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38522f51-cc0e-4026-a6f3-a9be0b429561-tls-certs\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:09.001518 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.001409    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-dshm\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:09.001518 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.001503    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-model-cache\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:09.001789 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.001532    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnw2n\" (UniqueName: \"kubernetes.io/projected/38522f51-cc0e-4026-a6f3-a9be0b429561-kube-api-access-bnw2n\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:09.001789 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.001586    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-home\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:09.001789 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.001633    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-tmp-dir\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:09.001789 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.001680    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38522f51-cc0e-4026-a6f3-a9be0b429561-tls-certs\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:09.002047 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.002024    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-model-cache\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:09.002345 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.002319    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-home\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:09.002566 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.002538    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-tmp-dir\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:09.004448 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.004429    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-dshm\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:09.004884 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.004862    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38522f51-cc0e-4026-a6f3-a9be0b429561-tls-certs\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:09.016788 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.016724    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnw2n\" (UniqueName: \"kubernetes.io/projected/38522f51-cc0e-4026-a6f3-a9be0b429561-kube-api-access-bnw2n\") pod \"distinct-w6-aabeef-kserve-84585dc6cf-xwcpj\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:09.112654 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.112559    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:09.415774 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:27:09.415730    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38522f51_cc0e_4026_a6f3_a9be0b429561.slice/crio-dfa7f9549d9aa11cedb473c1eb3a64b98bf41750163660cf3436dae8b2d8529a WatchSource:0}: Error finding container dfa7f9549d9aa11cedb473c1eb3a64b98bf41750163660cf3436dae8b2d8529a: Status 404 returned error can't find the container with id dfa7f9549d9aa11cedb473c1eb3a64b98bf41750163660cf3436dae8b2d8529a
Oct 31 13:27:09.416423 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.416392    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"]
Oct 31 13:27:09.967010 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.966945    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" event={"ID":"38522f51-cc0e-4026-a6f3-a9be0b429561","Type":"ContainerStarted","Data":"17027bb59555c60ed44298aeee010cc90e48bf0bb551d4d3a4828f7a41ec0175"}
Oct 31 13:27:09.967010 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.967004    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" event={"ID":"38522f51-cc0e-4026-a6f3-a9be0b429561","Type":"ContainerStarted","Data":"dfa7f9549d9aa11cedb473c1eb3a64b98bf41750163660cf3436dae8b2d8529a"}
Oct 31 13:27:09.994026 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:09.993977    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podStartSLOduration=1.993963573 podStartE2EDuration="1.993963573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:09.991118075 +0000 UTC m=+1429.609777241" watchObservedRunningTime="2026-10-31 13:27:09.993963573 +0000 UTC m=+1429.612622735"
Oct 31 13:27:10.113314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:10.112898    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:13.796830 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:13.796798    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"]
Oct 31 13:27:13.801210 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:13.801118    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:13.812329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:13.807826    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d85d82-openshift-default-dockercfg-xn9td\""
Oct 31 13:27:13.814680 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:13.814653    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"]
Oct 31 13:27:13.922001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:13.915872    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:13.922001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:13.915934    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x2np\" (UniqueName: \"kubernetes.io/projected/8906fc94-2d02-49d6-8ab0-c578e4b7391a-kube-api-access-2x2np\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:13.922001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:13.915979    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:13.922001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:13.916040    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:13.922001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:13.916081    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:13.922001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:13.916163    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:13.922001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:13.916199    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:13.922001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:13.916233    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:13.922001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:13.916261    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.017455 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.017397    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.017695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.017481    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.017695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.017515    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.017695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.017547    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.017695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.017575    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.017695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.017635    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.017695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.017674    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x2np\" (UniqueName: \"kubernetes.io/projected/8906fc94-2d02-49d6-8ab0-c578e4b7391a-kube-api-access-2x2np\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.018126 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.017709    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.018126 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.017777    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.018597 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.018571    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.018800 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.018775    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.018901 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.018882    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.019424 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.019396    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.020284 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.020254    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.024067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.024045    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.033526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.033502    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.039317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.037936    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x2np\" (UniqueName: \"kubernetes.io/projected/8906fc94-2d02-49d6-8ab0-c578e4b7391a-kube-api-access-2x2np\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.039317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.038808    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.165448 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.165405    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:14.460817 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.460791    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"]
Oct 31 13:27:14.461815 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:27:14.461787    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8906fc94_2d02_49d6_8ab0_c578e4b7391a.slice/crio-84cfca8a823127a666b1ad27986de9861787ca43379f6a107ec5b28ba89946a5 WatchSource:0}: Error finding container 84cfca8a823127a666b1ad27986de9861787ca43379f6a107ec5b28ba89946a5: Status 404 returned error can't find the container with id 84cfca8a823127a666b1ad27986de9861787ca43379f6a107ec5b28ba89946a5
Oct 31 13:27:14.464202 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.464176    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:27:14.464280 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.464233    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:27:14.464280 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:14.464261    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:27:15.028328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:15.026814    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6" event={"ID":"8906fc94-2d02-49d6-8ab0-c578e4b7391a","Type":"ContainerStarted","Data":"18098a7c79564dd9ba56e331626f241cf54a7087d09bf719b420dbd0f82a62e6"}
Oct 31 13:27:15.028328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:15.026919    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6" event={"ID":"8906fc94-2d02-49d6-8ab0-c578e4b7391a","Type":"ContainerStarted","Data":"84cfca8a823127a666b1ad27986de9861787ca43379f6a107ec5b28ba89946a5"}
Oct 31 13:27:15.055837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:15.055783    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6" podStartSLOduration=2.055770302 podStartE2EDuration="2.055770302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:15.052760888 +0000 UTC m=+1434.671420053" watchObservedRunningTime="2026-10-31 13:27:15.055770302 +0000 UTC m=+1434.674429466"
Oct 31 13:27:15.165678 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:15.165644    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:15.170443 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:15.170408    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 13:27:15.170569 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:15.170472    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6" podUID="8906fc94-2d02-49d6-8ab0-c578e4b7391a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 13:27:16.166283 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:16.166247    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 13:27:16.166792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:16.166325    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6" podUID="8906fc94-2d02-49d6-8ab0-c578e4b7391a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 13:27:17.166143 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:17.166107    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 13:27:17.166356 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:17.166179    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6" podUID="8906fc94-2d02-49d6-8ab0-c578e4b7391a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 13:27:18.166061 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:18.166019    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 13:27:18.166230 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:18.166092    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6" podUID="8906fc94-2d02-49d6-8ab0-c578e4b7391a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 13:27:19.114393 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:19.114356    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:19.148838 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:19.148807    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:19.166222 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:19.166190    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 13:27:19.166416 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:19.166248    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6" podUID="8906fc94-2d02-49d6-8ab0-c578e4b7391a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 13:27:20.043608 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.043569    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"]
Oct 31 13:27:20.053381 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.053345    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"]
Oct 31 13:27:20.053750 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.053671    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.069063 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.069035    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8b2f0a\"/\"distinct-w4-8b2f0a-kserve-self-signed-certs\""
Oct 31 13:27:20.090654 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.090628    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:27:20.166621 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.166581    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 13:27:20.167100 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.166644    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6" podUID="8906fc94-2d02-49d6-8ab0-c578e4b7391a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 13:27:20.188937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.188896    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f68b0a7c-08d8-42e2-8b56-f81b58386500-tls-certs\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.189094 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.189071    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-home\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.189148 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.189113    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-dshm\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.189188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.189173    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-model-cache\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.189357 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.189335    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-tmp-dir\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.189527 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.189502    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpmsh\" (UniqueName: \"kubernetes.io/projected/f68b0a7c-08d8-42e2-8b56-f81b58386500-kube-api-access-bpmsh\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.291183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.291140    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-tmp-dir\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.291388 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.291243    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpmsh\" (UniqueName: \"kubernetes.io/projected/f68b0a7c-08d8-42e2-8b56-f81b58386500-kube-api-access-bpmsh\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.291388 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.291304    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f68b0a7c-08d8-42e2-8b56-f81b58386500-tls-certs\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.291495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.291386    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-home\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.291495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.291413    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-dshm\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.291495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.291441    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-model-cache\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.291943 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.291917    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-model-cache\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.292198 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.292176    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-tmp-dir\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.292705 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.292684    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-home\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.295255 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.295201    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-dshm\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.295615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.295591    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f68b0a7c-08d8-42e2-8b56-f81b58386500-tls-certs\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.321018 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.320984    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpmsh\" (UniqueName: \"kubernetes.io/projected/f68b0a7c-08d8-42e2-8b56-f81b58386500-kube-api-access-bpmsh\") pod \"distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.376898 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.376859    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:20.692371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:20.692330    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"]
Oct 31 13:27:20.696232 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:27:20.696199    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf68b0a7c_08d8_42e2_8b56_f81b58386500.slice/crio-c4ec7ce38f57a235eaacbcd28361d22916db7ae9dce213e0d743255604f73158 WatchSource:0}: Error finding container c4ec7ce38f57a235eaacbcd28361d22916db7ae9dce213e0d743255604f73158: Status 404 returned error can't find the container with id c4ec7ce38f57a235eaacbcd28361d22916db7ae9dce213e0d743255604f73158
Oct 31 13:27:20.743612 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:27:20.742028    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:27:21.082216 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:21.082119    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" event={"ID":"f68b0a7c-08d8-42e2-8b56-f81b58386500","Type":"ContainerStarted","Data":"b84ba38848d2d9fd16e316b0fc0c62eae71410c020211c01e624c8e7a5d166e7"}
Oct 31 13:27:21.082216 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:21.082164    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" event={"ID":"f68b0a7c-08d8-42e2-8b56-f81b58386500","Type":"ContainerStarted","Data":"c4ec7ce38f57a235eaacbcd28361d22916db7ae9dce213e0d743255604f73158"}
Oct 31 13:27:21.107074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:21.107016    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podStartSLOduration=2.107004223 podStartE2EDuration="2.107004223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:21.105085061 +0000 UTC m=+1440.723744225" watchObservedRunningTime="2026-10-31 13:27:21.107004223 +0000 UTC m=+1440.725663387"
Oct 31 13:27:21.166261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:21.166220    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 13:27:21.166447 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:21.166282    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6" podUID="8906fc94-2d02-49d6-8ab0-c578e4b7391a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 13:27:21.377680 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:21.377643    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:22.171911 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:22.171882    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:22.172379 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:22.172353    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:22.173168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:22.173150    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:27:27.829983 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.829948    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"]
Oct 31 13:27:27.836637 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.836611    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:27.838698 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.838665    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"]
Oct 31 13:27:27.854462 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.853724    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6784d6\"/\"distinct-w5-6784d6-kserve-self-signed-certs\""
Oct 31 13:27:27.854462 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.853730    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6784d6\"/\"default-dockercfg-6gpxf\""
Oct 31 13:27:27.854462 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.853861    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6784d6\"/\"kube-root-ca.crt\""
Oct 31 13:27:27.854462 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.853906    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6784d6\"/\"openshift-service-ca.crt\""
Oct 31 13:27:27.891579 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.889971    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28afbd7d-8b18-44e6-8023-8a1019197ceb-tls-certs\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:27.891579 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.890034    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-home\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:27.891579 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.890061    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-tmp-dir\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:27.891579 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.890108    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-model-cache\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:27.891579 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.890153    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-dshm\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:27.891579 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.890187    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb4qj\" (UniqueName: \"kubernetes.io/projected/28afbd7d-8b18-44e6-8023-8a1019197ceb-kube-api-access-bb4qj\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:27.991033 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.990992    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-model-cache\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:27.992223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.992201    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-dshm\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:27.992787 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.992765    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb4qj\" (UniqueName: \"kubernetes.io/projected/28afbd7d-8b18-44e6-8023-8a1019197ceb-kube-api-access-bb4qj\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:27.992999 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.992983    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28afbd7d-8b18-44e6-8023-8a1019197ceb-tls-certs\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:27.993142 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.993127    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-home\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:27.993251 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.993236    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-tmp-dir\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:27.993397 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.992097    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-model-cache\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:27.993832 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.993811    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-home\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:27.993990 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:27.993966    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-tmp-dir\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:28.003994 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.003943    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28afbd7d-8b18-44e6-8023-8a1019197ceb-tls-certs\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:28.010698 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.010647    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb4qj\" (UniqueName: \"kubernetes.io/projected/28afbd7d-8b18-44e6-8023-8a1019197ceb-kube-api-access-bb4qj\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:28.018572 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.018540    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-dshm\") pod \"distinct-w5-6784d6-kserve-9747d98cc-8sh4w\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:28.194837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.194800    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:28.329933 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.327493    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"]
Oct 31 13:27:28.354429 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.347479    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"]
Oct 31 13:27:28.354429 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.347607    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.355734 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.354955    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7b3ddd-openshift-default-dockercfg-lzzgx\""
Oct 31 13:27:28.490628 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.490599    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"]
Oct 31 13:27:28.497385 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:27:28.493153    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28afbd7d_8b18_44e6_8023_8a1019197ceb.slice/crio-8f28a4470559bbe0e99f395f85272c8d84d59fc4ef15481a2df9536886cce8ae WatchSource:0}: Error finding container 8f28a4470559bbe0e99f395f85272c8d84d59fc4ef15481a2df9536886cce8ae: Status 404 returned error can't find the container with id 8f28a4470559bbe0e99f395f85272c8d84d59fc4ef15481a2df9536886cce8ae
Oct 31 13:27:28.503332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.497794    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c290e457-707d-4d9a-89e0-208ccb07111e-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.503332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.497955    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c290e457-707d-4d9a-89e0-208ccb07111e-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.503332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.497989    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv296\" (UniqueName: \"kubernetes.io/projected/c290e457-707d-4d9a-89e0-208ccb07111e-kube-api-access-sv296\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.503332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.498054    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c290e457-707d-4d9a-89e0-208ccb07111e-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.503332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.498091    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.503332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.498120    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.503332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.498136    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.503332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.498278    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.503332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.498373    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.599361 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.599284    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.599549 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.599379    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.599549 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.599449    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c290e457-707d-4d9a-89e0-208ccb07111e-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.599660 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.599555    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c290e457-707d-4d9a-89e0-208ccb07111e-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.599660 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.599598    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv296\" (UniqueName: \"kubernetes.io/projected/c290e457-707d-4d9a-89e0-208ccb07111e-kube-api-access-sv296\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.599660 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.599631    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c290e457-707d-4d9a-89e0-208ccb07111e-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.599807 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.599742    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.599807 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.599787    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.599909 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.599823    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.600180 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.600159    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.600394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.600379    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.601138 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.601118    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c290e457-707d-4d9a-89e0-208ccb07111e-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.607110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.603261    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.607110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.603597    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.607110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.604757    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"]
Oct 31 13:27:28.607110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.606286    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c290e457-707d-4d9a-89e0-208ccb07111e-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.609632 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.609611    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.609958 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.609938    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.623628 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.623603    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8b2f0a-openshift-default-dockercfg-w6shk\""
Oct 31 13:27:28.626779 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.626421    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv296\" (UniqueName: \"kubernetes.io/projected/c290e457-707d-4d9a-89e0-208ccb07111e-kube-api-access-sv296\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.627871 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.627807    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c290e457-707d-4d9a-89e0-208ccb07111e-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.629039 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.628765    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"]
Oct 31 13:27:28.672559 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.672517    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:28.802269 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.802233    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.802445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.802391    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.802497 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.802448    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.802605 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.802583    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.802655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.802628    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.802768 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.802746    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040f7f90-43a7-4162-8de4-d17a85cb15ca-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.802829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.802792    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s45lx\" (UniqueName: \"kubernetes.io/projected/040f7f90-43a7-4162-8de4-d17a85cb15ca-kube-api-access-s45lx\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.802884 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.802842    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.802884 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.802874    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.903707 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.903672    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040f7f90-43a7-4162-8de4-d17a85cb15ca-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.903707 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.903712    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s45lx\" (UniqueName: \"kubernetes.io/projected/040f7f90-43a7-4162-8de4-d17a85cb15ca-kube-api-access-s45lx\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.904236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.903742    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.904236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.903864    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.904236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.903943    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.904236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.904037    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.904236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.904082    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.904236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.904171    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.904586 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.904355    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.904586 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.904533    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.904586 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.904554    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.904734 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.904627    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.904734 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.904663    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040f7f90-43a7-4162-8de4-d17a85cb15ca-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.904849 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.904827    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.906224 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.906198    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.907025 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.907002    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.916656 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.916633    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s45lx\" (UniqueName: \"kubernetes.io/projected/040f7f90-43a7-4162-8de4-d17a85cb15ca-kube-api-access-s45lx\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.916749 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.916669    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.939552 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.939531    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:28.967402 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.967378    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"]
Oct 31 13:27:28.968241 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:27:28.968218    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc290e457_707d_4d9a_89e0_208ccb07111e.slice/crio-75642706d8896974ee9a75cc3f5d75c105e8b2d3730ee3c99ab4b4060b15e862 WatchSource:0}: Error finding container 75642706d8896974ee9a75cc3f5d75c105e8b2d3730ee3c99ab4b4060b15e862: Status 404 returned error can't find the container with id 75642706d8896974ee9a75cc3f5d75c105e8b2d3730ee3c99ab4b4060b15e862
Oct 31 13:27:28.971236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.971200    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:27:28.971348 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.971271    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:27:28.971348 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:28.971323    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:27:29.159052 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:29.159011    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp" event={"ID":"c290e457-707d-4d9a-89e0-208ccb07111e","Type":"ContainerStarted","Data":"1d88cb997d2eef40bb20ce84623c6f5169edec43e34aa73588a701b633cf723f"}
Oct 31 13:27:29.159274 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:29.159117    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp" event={"ID":"c290e457-707d-4d9a-89e0-208ccb07111e","Type":"ContainerStarted","Data":"75642706d8896974ee9a75cc3f5d75c105e8b2d3730ee3c99ab4b4060b15e862"}
Oct 31 13:27:29.161779 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:29.161751    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" event={"ID":"28afbd7d-8b18-44e6-8023-8a1019197ceb","Type":"ContainerStarted","Data":"5d32dc720c70379446093b674ea661e2135d4e36c673e8c11926917141fd5e2f"}
Oct 31 13:27:29.161884 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:29.161789    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" event={"ID":"28afbd7d-8b18-44e6-8023-8a1019197ceb","Type":"ContainerStarted","Data":"8f28a4470559bbe0e99f395f85272c8d84d59fc4ef15481a2df9536886cce8ae"}
Oct 31 13:27:29.189819 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:29.189773    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp" podStartSLOduration=1.189760754 podStartE2EDuration="1.189760754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:29.187790397 +0000 UTC m=+1448.806449562" watchObservedRunningTime="2026-10-31 13:27:29.189760754 +0000 UTC m=+1448.808419918"
Oct 31 13:27:29.194914 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:29.194890    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:29.211688 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:29.211638    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podStartSLOduration=2.211620178 podStartE2EDuration="2.211620178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:29.208409905 +0000 UTC m=+1448.827069067" watchObservedRunningTime="2026-10-31 13:27:29.211620178 +0000 UTC m=+1448.830279343"
Oct 31 13:27:29.235885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:29.235856    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"]
Oct 31 13:27:29.237807 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:27:29.237785    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040f7f90_43a7_4162_8de4_d17a85cb15ca.slice/crio-b8bcf7086aba2ab61da355341d418d0058a492bf2b5b79fcef6adb8536346243 WatchSource:0}: Error finding container b8bcf7086aba2ab61da355341d418d0058a492bf2b5b79fcef6adb8536346243: Status 404 returned error can't find the container with id b8bcf7086aba2ab61da355341d418d0058a492bf2b5b79fcef6adb8536346243
Oct 31 13:27:29.240654 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:29.240626    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:27:29.240762 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:29.240685    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:27:29.240762 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:29.240714    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:27:29.672722 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:29.672684    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:30.170833 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:30.170796    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66" event={"ID":"040f7f90-43a7-4162-8de4-d17a85cb15ca","Type":"ContainerStarted","Data":"581236f086a6ab6a23cb40b8440f34fcf86efc2898354327b563c6f577646f28"}
Oct 31 13:27:30.171333 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:30.170851    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66" event={"ID":"040f7f90-43a7-4162-8de4-d17a85cb15ca","Type":"ContainerStarted","Data":"b8bcf7086aba2ab61da355341d418d0058a492bf2b5b79fcef6adb8536346243"}
Oct 31 13:27:30.196996 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:30.196948    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66" podStartSLOduration=2.196936044 podStartE2EDuration="2.196936044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:30.194799593 +0000 UTC m=+1449.813458759" watchObservedRunningTime="2026-10-31 13:27:30.196936044 +0000 UTC m=+1449.815595207"
Oct 31 13:27:30.377785 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:30.377750    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:30.392405 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:30.392381    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:30.682320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:30.679450    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 13:27:30.682320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:30.679512    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp" podUID="c290e457-707d-4d9a-89e0-208ccb07111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 13:27:30.940667 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:30.940467    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:30.943036 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:30.942997    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:27:30.943284 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:30.943251    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66" podUID="040f7f90-43a7-4162-8de4-d17a85cb15ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 13:27:31.188416 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:31.188391    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:27:31.674686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:31.674647    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 13:27:31.674892 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:31.674702    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp" podUID="c290e457-707d-4d9a-89e0-208ccb07111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 13:27:31.946369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:31.945675    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:27:31.946369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:31.945736    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66" podUID="040f7f90-43a7-4162-8de4-d17a85cb15ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 13:27:32.674670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:32.674634    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 13:27:32.675213 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:32.674696    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp" podUID="c290e457-707d-4d9a-89e0-208ccb07111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 13:27:32.945766 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:32.945666    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:27:32.946043 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:32.946015    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66" podUID="040f7f90-43a7-4162-8de4-d17a85cb15ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 13:27:33.674318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:33.673622    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 13:27:33.674318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:33.673691    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp" podUID="c290e457-707d-4d9a-89e0-208ccb07111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 13:27:33.949409 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:33.944644    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:27:33.949409 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:33.944706    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66" podUID="040f7f90-43a7-4162-8de4-d17a85cb15ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 13:27:34.673845 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:34.673805    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 13:27:34.674030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:34.673858    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp" podUID="c290e457-707d-4d9a-89e0-208ccb07111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 13:27:34.947373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:34.946803    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:27:34.947373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:34.946881    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66" podUID="040f7f90-43a7-4162-8de4-d17a85cb15ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 13:27:35.673369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:35.673325    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 13:27:35.673860 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:35.673402    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp" podUID="c290e457-707d-4d9a-89e0-208ccb07111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 13:27:35.947797 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:35.947701    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:27:35.948078 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:35.948046    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66" podUID="040f7f90-43a7-4162-8de4-d17a85cb15ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 13:27:36.679130 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:36.679091    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:36.679716 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:36.679693    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:36.680570 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:36.680529    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:27:37.172507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:37.172476    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:37.172918 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:37.172894    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:37.174042 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:37.174017    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:27:38.195013 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:38.194984    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:38.209578 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:38.209551    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:38.256395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:38.256361    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:27:41.382833 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.382794    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"]
Oct 31 13:27:41.387113 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.387089    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.392757 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.392736    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5286ea\"/\"distinct-w2-5286ea-kserve-self-signed-certs\""
Oct 31 13:27:41.392941 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.392918    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5286ea\"/\"openshift-service-ca.crt\""
Oct 31 13:27:41.393050 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.392951    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5286ea\"/\"default-dockercfg-r7fxl\""
Oct 31 13:27:41.393118 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.392985    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5286ea\"/\"kube-root-ca.crt\""
Oct 31 13:27:41.404350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.402305    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"]
Oct 31 13:27:41.511763 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.511704    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8742e18-cae9-470f-8279-a78c237a4c58-tls-certs\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.512175 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.512154    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-tmp-dir\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.512357 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.512339    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-model-cache\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.512551 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.512535    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-home\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.512717 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.512703    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-dshm\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.512860 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.512847    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzrtr\" (UniqueName: \"kubernetes.io/projected/a8742e18-cae9-470f-8279-a78c237a4c58-kube-api-access-lzrtr\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.614169 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.614127    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-tmp-dir\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.614169 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.614179    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-model-cache\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.614485 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.614265    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-home\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.614485 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.614336    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-dshm\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.614485 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.614376    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrtr\" (UniqueName: \"kubernetes.io/projected/a8742e18-cae9-470f-8279-a78c237a4c58-kube-api-access-lzrtr\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.614650 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.614551    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8742e18-cae9-470f-8279-a78c237a4c58-tls-certs\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.614761 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.614738    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-tmp-dir\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.615253 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.615222    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-model-cache\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.615703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.615679    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-home\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.618323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.618281    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-dshm\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.619082 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.619057    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8742e18-cae9-470f-8279-a78c237a4c58-tls-certs\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.629646 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.629620    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzrtr\" (UniqueName: \"kubernetes.io/projected/a8742e18-cae9-470f-8279-a78c237a4c58-kube-api-access-lzrtr\") pod \"distinct-w2-5286ea-kserve-9bdd7d864-qfrbk\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:41.710104 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:41.710018    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:42.008236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:42.008207    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"]
Oct 31 13:27:42.009103 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:27:42.009072    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8742e18_cae9_470f_8279_a78c237a4c58.slice/crio-81295f2b57afd187e9a75292a08f7616e12beab01e5778073c621f3e0d2f590c WatchSource:0}: Error finding container 81295f2b57afd187e9a75292a08f7616e12beab01e5778073c621f3e0d2f590c: Status 404 returned error can't find the container with id 81295f2b57afd187e9a75292a08f7616e12beab01e5778073c621f3e0d2f590c
Oct 31 13:27:42.274098 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:42.274016    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" event={"ID":"a8742e18-cae9-470f-8279-a78c237a4c58","Type":"ContainerStarted","Data":"7d67759a957948021aa2a55a26421e06d14745abe1dc081365c24a10ffab849c"}
Oct 31 13:27:42.274098 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:42.274058    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" event={"ID":"a8742e18-cae9-470f-8279-a78c237a4c58","Type":"ContainerStarted","Data":"81295f2b57afd187e9a75292a08f7616e12beab01e5778073c621f3e0d2f590c"}
Oct 31 13:27:42.296919 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:42.296872    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podStartSLOduration=1.2968583900000001 podStartE2EDuration="1.29685839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:42.294229853 +0000 UTC m=+1461.912889028" watchObservedRunningTime="2026-10-31 13:27:42.29685839 +0000 UTC m=+1461.915517554"
Oct 31 13:27:42.710992 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:42.710956    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:43.318042 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.318010    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"]
Oct 31 13:27:43.322907 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.322888    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.328446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.328425    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-6784d6-openshift-default-dockercfg-xbv85\""
Oct 31 13:27:43.338741 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.338717    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"]
Oct 31 13:27:43.438213 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.438177    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.438439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.438231    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.438439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.438399    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd5f9\" (UniqueName: \"kubernetes.io/projected/13e81951-495d-4169-bdbb-f6484e5d4756-kube-api-access-zd5f9\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.438569 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.438477    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.438569 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.438563    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.438681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.438660    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e81951-495d-4169-bdbb-f6484e5d4756-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.438725 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.438706    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e81951-495d-4169-bdbb-f6484e5d4756-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.438802 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.438784    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.438864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.438834    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e81951-495d-4169-bdbb-f6484e5d4756-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.540366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.540338    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e81951-495d-4169-bdbb-f6484e5d4756-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.540560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.540378    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e81951-495d-4169-bdbb-f6484e5d4756-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.540560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.540408    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.540560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.540434    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e81951-495d-4169-bdbb-f6484e5d4756-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.540560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.540499    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.540560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.540520    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.540560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.540558    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd5f9\" (UniqueName: \"kubernetes.io/projected/13e81951-495d-4169-bdbb-f6484e5d4756-kube-api-access-zd5f9\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.540851 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.540623    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.540851 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.540677    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.540972 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.540886    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.541051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.541034    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.541191 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.541166    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.541472 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.541453    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.541831 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.541807    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e81951-495d-4169-bdbb-f6484e5d4756-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.542999 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.542974    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.543636 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.543565    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e81951-495d-4169-bdbb-f6484e5d4756-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.553804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.553782    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e81951-495d-4169-bdbb-f6484e5d4756-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.553804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.553795    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd5f9\" (UniqueName: \"kubernetes.io/projected/13e81951-495d-4169-bdbb-f6484e5d4756-kube-api-access-zd5f9\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.641446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.641417    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:43.941193 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:27:43.941158    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e81951_495d_4169_bdbb_f6484e5d4756.slice/crio-56c6187b609457ce77cf3fd6ec150a64d15cf2b320bd310a49e1159f1d212609 WatchSource:0}: Error finding container 56c6187b609457ce77cf3fd6ec150a64d15cf2b320bd310a49e1159f1d212609: Status 404 returned error can't find the container with id 56c6187b609457ce77cf3fd6ec150a64d15cf2b320bd310a49e1159f1d212609
Oct 31 13:27:43.943761 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.943735    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"]
Oct 31 13:27:43.944511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.944475    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:27:43.944575 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.944555    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:27:43.944613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:43.944594    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:27:44.291798 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:44.291701    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs" event={"ID":"13e81951-495d-4169-bdbb-f6484e5d4756","Type":"ContainerStarted","Data":"d2b3f22c9fa201de3384b5beabbff18dce88667ca3462d7fecc720744cc758ca"}
Oct 31 13:27:44.291798 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:44.291760    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs" event={"ID":"13e81951-495d-4169-bdbb-f6484e5d4756","Type":"ContainerStarted","Data":"56c6187b609457ce77cf3fd6ec150a64d15cf2b320bd310a49e1159f1d212609"}
Oct 31 13:27:44.323310 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:44.323241    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs" podStartSLOduration=1.323221907 podStartE2EDuration="1.323221907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:44.319639761 +0000 UTC m=+1463.938298927" watchObservedRunningTime="2026-10-31 13:27:44.323221907 +0000 UTC m=+1463.941881073"
Oct 31 13:27:44.645431 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:44.645389    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:45.657054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:45.648453    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:27:45.657054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:45.648517    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs" podUID="13e81951-495d-4169-bdbb-f6484e5d4756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 13:27:46.644318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:46.644152    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:27:46.644318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:46.644217    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs" podUID="13e81951-495d-4169-bdbb-f6484e5d4756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 13:27:47.644608 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:47.644568    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:27:47.645197 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:47.645160    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs" podUID="13e81951-495d-4169-bdbb-f6484e5d4756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 13:27:48.642872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:48.642833    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:27:48.643054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:48.642905    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs" podUID="13e81951-495d-4169-bdbb-f6484e5d4756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 13:27:49.651690 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:49.651650    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:27:49.652253 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:49.652219    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs" podUID="13e81951-495d-4169-bdbb-f6484e5d4756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 13:27:50.642187 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:50.642150    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:27:50.642359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:50.642217    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs" podUID="13e81951-495d-4169-bdbb-f6484e5d4756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 13:27:50.738517 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:27:50.738480    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:27:51.712036 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:51.712003    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:51.732803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:51.732770    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:52.207064 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:52.206267    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:52.207497 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:52.207321    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:52.208125 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:52.208101    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:27:52.416572 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:52.416536    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:27:58.718428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.718382    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"]
Oct 31 13:27:58.724764 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.724737    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.737872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.737844    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-aabeef-openshift-default-dockercfg-wlnkc\""
Oct 31 13:27:58.740795 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.740771    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"]
Oct 31 13:27:58.792472 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.792443    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.792636 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.792482    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.792636 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.792588    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.792760 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.792649    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.792760 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.792725    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.792897 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.792780    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.792897 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.792873    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5dqr\" (UniqueName: \"kubernetes.io/projected/5c132a2a-1e7e-4858-80fb-376083a5b0f7-kube-api-access-r5dqr\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.792962 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.792906    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.792962 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.792957    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.894595 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.894559    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.894765 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.894609    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.894765 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.894637    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.894765 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.894670    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.894765 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.894703    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5dqr\" (UniqueName: \"kubernetes.io/projected/5c132a2a-1e7e-4858-80fb-376083a5b0f7-kube-api-access-r5dqr\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.894765 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.894719    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.895041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.894795    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.895041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.894861    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.895041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.894896    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.895204 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.895110    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.895204 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.895162    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.895346 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.895249    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.895346 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.895310    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.895591 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.895570    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.896817 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.896796    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.897270 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.897253    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.916843 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.916817    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:58.917758 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:58.917725    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5dqr\" (UniqueName: \"kubernetes.io/projected/5c132a2a-1e7e-4858-80fb-376083a5b0f7-kube-api-access-r5dqr\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:59.042755 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:59.042672    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:27:59.335732 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:59.335701    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"]
Oct 31 13:27:59.337086 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:27:59.337059    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c132a2a_1e7e_4858_80fb_376083a5b0f7.slice/crio-c6c865e14ca66768a5eeccee7a0fdb0306edb027fef6f2b9cc0ec0b090f610f6 WatchSource:0}: Error finding container c6c865e14ca66768a5eeccee7a0fdb0306edb027fef6f2b9cc0ec0b090f610f6: Status 404 returned error can't find the container with id c6c865e14ca66768a5eeccee7a0fdb0306edb027fef6f2b9cc0ec0b090f610f6
Oct 31 13:27:59.339709 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:59.339671    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:27:59.339808 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:59.339757    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:27:59.339861 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:59.339827    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:27:59.462687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:59.462649    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg" event={"ID":"5c132a2a-1e7e-4858-80fb-376083a5b0f7","Type":"ContainerStarted","Data":"761ebd16829fb399522f86d68a5bbdff757a5961e5ae32f5ddcaaa6d8d1ce17b"}
Oct 31 13:27:59.462861 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:59.462711    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg" event={"ID":"5c132a2a-1e7e-4858-80fb-376083a5b0f7","Type":"ContainerStarted","Data":"c6c865e14ca66768a5eeccee7a0fdb0306edb027fef6f2b9cc0ec0b090f610f6"}
Oct 31 13:27:59.491067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:27:59.491004    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg" podStartSLOduration=1.490985075 podStartE2EDuration="1.490985075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:59.487057964 +0000 UTC m=+1479.105717129" watchObservedRunningTime="2026-10-31 13:27:59.490985075 +0000 UTC m=+1479.109644239"
Oct 31 13:28:00.042966 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:00.042878    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:28:00.044326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:00.044280    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 13:28:00.044425 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:00.044368    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg" podUID="5c132a2a-1e7e-4858-80fb-376083a5b0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 13:28:01.043458 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:01.043420    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 13:28:01.043842 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:01.043487    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg" podUID="5c132a2a-1e7e-4858-80fb-376083a5b0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 13:28:02.043720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:02.043685    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 13:28:02.044326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:02.044272    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg" podUID="5c132a2a-1e7e-4858-80fb-376083a5b0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 13:28:03.043348 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:03.043278    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 13:28:03.043540 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:03.043433    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg" podUID="5c132a2a-1e7e-4858-80fb-376083a5b0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 13:28:04.043194 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:04.043157    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 13:28:04.043668 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:04.043223    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg" podUID="5c132a2a-1e7e-4858-80fb-376083a5b0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 13:28:05.043799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:05.043758    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 13:28:05.044427 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:05.044391    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg" podUID="5c132a2a-1e7e-4858-80fb-376083a5b0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 13:28:06.046723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:06.046683    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 13:28:06.047412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:06.047381    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg" podUID="5c132a2a-1e7e-4858-80fb-376083a5b0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 13:28:08.099318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:08.097389    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:28:08.099318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:08.097502    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg" podUID="5c132a2a-1e7e-4858-80fb-376083a5b0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:08.236368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:08.236330    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:28:08.236689 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:08.236637    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66" podUID="040f7f90-43a7-4162-8de4-d17a85cb15ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:08.471435 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:08.471264    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:28:08.474334 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:08.471912    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:28:08.474334 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:08.473003    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:28:09.525493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.524173    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"]
Oct 31 13:28:09.697398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.668616    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"]
Oct 31 13:28:09.697398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.668736    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.697398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.681487    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7b3ddd\"/\"distinct-w0-7b3ddd-kserve-self-signed-certs\""
Oct 31 13:28:09.752743 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.752705    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-home\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.753098 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.753073    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-dshm\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.753596 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.753573    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64ea0e4-ea86-497c-b24c-f8326ebe340b-tls-certs\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.754078 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.754060    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-tmp-dir\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.754324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.754285    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-model-cache\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.754723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.754704    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5vjn\" (UniqueName: \"kubernetes.io/projected/e64ea0e4-ea86-497c-b24c-f8326ebe340b-kube-api-access-m5vjn\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.858004 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.857894    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-model-cache\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.858168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.858023    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5vjn\" (UniqueName: \"kubernetes.io/projected/e64ea0e4-ea86-497c-b24c-f8326ebe340b-kube-api-access-m5vjn\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.858168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.858106    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-home\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.858344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.858188    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-dshm\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.858344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.858238    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64ea0e4-ea86-497c-b24c-f8326ebe340b-tls-certs\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.858344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.858316    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-tmp-dir\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.858505 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.858370    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-model-cache\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.858915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.858891    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-tmp-dir\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.859068 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.858933    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-home\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.863114 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.861695    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-dshm\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.863975 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.863954    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64ea0e4-ea86-497c-b24c-f8326ebe340b-tls-certs\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:09.871971 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:09.871949    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5vjn\" (UniqueName: \"kubernetes.io/projected/e64ea0e4-ea86-497c-b24c-f8326ebe340b-kube-api-access-m5vjn\") pod \"distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:10.043314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:10.043185    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:10.352394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:10.352366    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"]
Oct 31 13:28:10.354708 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:28:10.354682    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode64ea0e4_ea86_497c_b24c_f8326ebe340b.slice/crio-f110a245e7bc6367fa1e3b507b775882ba7e4feb86dd626631adf264dc793193 WatchSource:0}: Error finding container f110a245e7bc6367fa1e3b507b775882ba7e4feb86dd626631adf264dc793193: Status 404 returned error can't find the container with id f110a245e7bc6367fa1e3b507b775882ba7e4feb86dd626631adf264dc793193
Oct 31 13:28:10.682377 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:10.682283    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" event={"ID":"e64ea0e4-ea86-497c-b24c-f8326ebe340b","Type":"ContainerStarted","Data":"aac5c2cbfc08786fd20489700311a152f01770c0ebb2ca1caba6242d1c927fad"}
Oct 31 13:28:10.682377 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:10.682347    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" event={"ID":"e64ea0e4-ea86-497c-b24c-f8326ebe340b","Type":"ContainerStarted","Data":"f110a245e7bc6367fa1e3b507b775882ba7e4feb86dd626631adf264dc793193"}
Oct 31 13:28:10.704706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:10.704660    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podStartSLOduration=1.7046477150000001 podStartE2EDuration="1.704647715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:10.702927894 +0000 UTC m=+1490.321587058" watchObservedRunningTime="2026-10-31 13:28:10.704647715 +0000 UTC m=+1490.323306878"
Oct 31 13:28:11.043702 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:11.043624    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:20.044285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.044243    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:20.060462 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.060437    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:20.431703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.431671    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"]
Oct 31 13:28:20.438024 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.437997    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.446429 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.446399    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"]
Oct 31 13:28:20.449561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.448759    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5286ea\"/\"distinct-2-w2-5286ea-kserve-self-signed-certs\""
Oct 31 13:28:20.504665 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.504624    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-tmp-dir\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.504853 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.504682    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-home\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.504853 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.504731    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-dshm\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.504853 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.504825    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddwq5\" (UniqueName: \"kubernetes.io/projected/20570d2d-2c07-487b-a561-59e10e351154-kube-api-access-ddwq5\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.505030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.504881    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20570d2d-2c07-487b-a561-59e10e351154-tls-certs\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.505030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.504933    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-model-cache\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.605753 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.605704    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddwq5\" (UniqueName: \"kubernetes.io/projected/20570d2d-2c07-487b-a561-59e10e351154-kube-api-access-ddwq5\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.606060 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.606026    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20570d2d-2c07-487b-a561-59e10e351154-tls-certs\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.606185 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.606101    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-model-cache\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.606185 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.606171    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-tmp-dir\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.606381 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.606209    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-home\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.606381 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.606267    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-dshm\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.606729 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.606703    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-model-cache\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.607582 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.607556    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-tmp-dir\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.607818 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.607797    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-home\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.609588 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.609550    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5286ea\"/\"distinct-2-w2-5286ea-kserve-self-signed-certs\""
Oct 31 13:28:20.622488 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.621703    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-dshm\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.636836 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.636809    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20570d2d-2c07-487b-a561-59e10e351154-tls-certs\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.637027 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.636915    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddwq5\" (UniqueName: \"kubernetes.io/projected/20570d2d-2c07-487b-a561-59e10e351154-kube-api-access-ddwq5\") pod \"distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") " pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.728316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.728225    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:28:20.731167 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.731138    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:28:20.738716 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.738697    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:28:20.741599 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.741581    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:28:20.746347 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:28:20.746317    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:28:20.761430 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.761406    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:28:20.764673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.764652    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:20.767118 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.767099    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:28:20.776070 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:20.776047    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:28:21.094743 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:21.094662    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"]
Oct 31 13:28:21.115950 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:28:21.115911    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20570d2d_2c07_487b_a561_59e10e351154.slice/crio-f15c6533315f40ebe87d781911a31b1649846cb041eeec0cfb9f21a38314ff84 WatchSource:0}: Error finding container f15c6533315f40ebe87d781911a31b1649846cb041eeec0cfb9f21a38314ff84: Status 404 returned error can't find the container with id f15c6533315f40ebe87d781911a31b1649846cb041eeec0cfb9f21a38314ff84
Oct 31 13:28:21.767445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:21.767410    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" event={"ID":"20570d2d-2c07-487b-a561-59e10e351154","Type":"ContainerStarted","Data":"16d276aaa0f3fef64b2b5a8fd76a436423af30df7cb05f31423ee0f092a6acf1"}
Oct 31 13:28:21.767445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:21.767458    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" event={"ID":"20570d2d-2c07-487b-a561-59e10e351154","Type":"ContainerStarted","Data":"f15c6533315f40ebe87d781911a31b1649846cb041eeec0cfb9f21a38314ff84"}
Oct 31 13:28:21.788028 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:21.787964    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podStartSLOduration=1.787947199 podStartE2EDuration="1.787947199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:21.787746396 +0000 UTC m=+1501.406405561" watchObservedRunningTime="2026-10-31 13:28:21.787947199 +0000 UTC m=+1501.406606363"
Oct 31 13:28:22.765880 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:22.765846    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:28.614319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.613393    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"]
Oct 31 13:28:28.689525 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.686068    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:28.689525 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.689318    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"]
Oct 31 13:28:28.789335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.767044    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:28.789335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.767137    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:28.789335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.767174    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:28.789335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.767199    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a36003d-cda6-447e-bb9f-95f13e32c276-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:28.789335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.767234    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:28.789335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.767271    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:28.789335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.767318    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrlg2\" (UniqueName: \"kubernetes.io/projected/6a36003d-cda6-447e-bb9f-95f13e32c276-kube-api-access-rrlg2\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:28.789335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.767352    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:28.789335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.767378    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:28.905360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.886108    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:28.905360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.886194    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:28.905360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.886220    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a36003d-cda6-447e-bb9f-95f13e32c276-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:28.905360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.886274    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:28.905360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.886343    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:28.905360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:28.886391    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrlg2\" (UniqueName: \"kubernetes.io/projected/6a36003d-cda6-447e-bb9f-95f13e32c276-kube-api-access-rrlg2\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:29.095319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.095002    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:29.095319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.095088    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:29.095568 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.095332    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:29.187373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.187184    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:29.210316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.205604    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"]
Oct 31 13:28:29.236436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.235259    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:29.236436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.236340    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:29.236678 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.236644    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:29.237820 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.236814    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:29.249620 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.246704    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a36003d-cda6-447e-bb9f-95f13e32c276-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:29.249620 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.247600    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.315313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.309008    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:29.315313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.311799    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:29.315313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.312959    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrlg2\" (UniqueName: \"kubernetes.io/projected/6a36003d-cda6-447e-bb9f-95f13e32c276-kube-api-access-rrlg2\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:29.315313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.313022    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"]
Oct 31 13:28:29.385366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.385026    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"]
Oct 31 13:28:29.389313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.389008    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:29.391770 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.390491    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.422628 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.422596    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"]
Oct 31 13:28:29.426724 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.426702    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5286ea-openshift-default-dockercfg-rsjlf\""
Oct 31 13:28:29.453012 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.452059    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.453264 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.453244    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64a3375-fe5e-426f-bc22-bcaca79b2722-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.453422 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.453400    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.469368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.465964    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.469368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.466077    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.469368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.466112    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.469368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.466143    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.469368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.466170    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.469368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.466225    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.469368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.466259    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.469368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.466322    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.469368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.466373    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.469368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.466445    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.469368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.466499    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.469368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.466548    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hggg\" (UniqueName: \"kubernetes.io/projected/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-kube-api-access-5hggg\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.469368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.466606    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.469368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.466634    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.469368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.466675    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd2r7\" (UniqueName: \"kubernetes.io/projected/e64a3375-fe5e-426f-bc22-bcaca79b2722-kube-api-access-wd2r7\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.577316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.577184    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.577316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.577311    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.577595 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.577360    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.577595 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.577416    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hggg\" (UniqueName: \"kubernetes.io/projected/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-kube-api-access-5hggg\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.577595 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.577481    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.577595 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.577509    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.577595 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.577562    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd2r7\" (UniqueName: \"kubernetes.io/projected/e64a3375-fe5e-426f-bc22-bcaca79b2722-kube-api-access-wd2r7\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.577887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.577641    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.577887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.577719    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64a3375-fe5e-426f-bc22-bcaca79b2722-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.577887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.577770    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.577887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.577801    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.577887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.577848    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.577887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.577876    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.578190 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.577900    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.578190 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.577937    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.578190 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.578004    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.578190 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.578031    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.578190 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.578086    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.595109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.578729    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.595109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.579088    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.595109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.582360    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64a3375-fe5e-426f-bc22-bcaca79b2722-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.595109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.582731    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.595109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.582986    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.595109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.584130    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.595109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.589594    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.595109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.589937    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.595109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.591026    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.595109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.591263    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.595109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.591514    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.620548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.612882    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.620548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.613599    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.620548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.618642    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hggg\" (UniqueName: \"kubernetes.io/projected/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-kube-api-access-5hggg\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.621115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.620840    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.628753 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.628703    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.632320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.631735    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:29.632320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.632260    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd2r7\" (UniqueName: \"kubernetes.io/projected/e64a3375-fe5e-426f-bc22-bcaca79b2722-kube-api-access-wd2r7\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.776766 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.776561    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:29.825105 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:29.824516    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:30.047726 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.047580    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"]
Oct 31 13:28:30.178150 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.166589    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:28:30.178150 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.166694    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:28:30.178150 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.166746    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:28:30.471725 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.469528    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"]
Oct 31 13:28:30.471872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.471831    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"]
Oct 31 13:28:30.476618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.476572    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:28:30.476811 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.476787    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:28:30.476969 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.476949    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:28:30.477076 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:28:30.477039    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64a3375_fe5e_426f_bc22_bcaca79b2722.slice/crio-06ed7abaf91f20ae7b2d19d4ed6a9bcd85e0dd362f5f46ea374aada3f34e2ae6 WatchSource:0}: Error finding container 06ed7abaf91f20ae7b2d19d4ed6a9bcd85e0dd362f5f46ea374aada3f34e2ae6: Status 404 returned error can't find the container with id 06ed7abaf91f20ae7b2d19d4ed6a9bcd85e0dd362f5f46ea374aada3f34e2ae6
Oct 31 13:28:30.481594 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.481561    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:28:30.481685 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.481633    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:28:30.481685 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.481676    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:28:30.767275 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.766357    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:30.791885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.791857    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:30.978652 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.978619    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j" event={"ID":"6a36003d-cda6-447e-bb9f-95f13e32c276","Type":"ContainerStarted","Data":"0ddafa001e3e28c27e8f86d931ed37ee0e22eed1ea065d4bbfbbe17bdcb5afaf"}
Oct 31 13:28:30.978866 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.978688    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j" event={"ID":"6a36003d-cda6-447e-bb9f-95f13e32c276","Type":"ContainerStarted","Data":"2748aa8f1f91707746ccd3de41fe95b2c567c8e3438ad76912e0892e965d1621"}
Oct 31 13:28:30.981693 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.981003    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" event={"ID":"e64a3375-fe5e-426f-bc22-bcaca79b2722","Type":"ContainerStarted","Data":"cdf87235268786cf2bc510827622283b3d5f06ad016a2cd81c385b3bc5be62f5"}
Oct 31 13:28:30.981693 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.981057    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" event={"ID":"e64a3375-fe5e-426f-bc22-bcaca79b2722","Type":"ContainerStarted","Data":"06ed7abaf91f20ae7b2d19d4ed6a9bcd85e0dd362f5f46ea374aada3f34e2ae6"}
Oct 31 13:28:30.983280 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.983250    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj" event={"ID":"11d0989b-0ce8-4041-bf02-e9af6b10c3ae","Type":"ContainerStarted","Data":"117a3b16d6b5851cd1cae61ecf0c84b5e0f2f7acc73a32f5b93946d4c0c0096a"}
Oct 31 13:28:30.983422 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.983317    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj" event={"ID":"11d0989b-0ce8-4041-bf02-e9af6b10c3ae","Type":"ContainerStarted","Data":"a55bd516c828aa051c804c314029fe06d6f84c9c07def42d46f0db01a9ba35f9"}
Oct 31 13:28:30.996472 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:30.996441    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:28:31.011126 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:31.011054    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j" podStartSLOduration=3.010986627 podStartE2EDuration="3.010986627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:31.007845344 +0000 UTC m=+1510.626504525" watchObservedRunningTime="2026-10-31 13:28:31.010986627 +0000 UTC m=+1510.629645792"
Oct 31 13:28:31.044430 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:31.044378    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj" podStartSLOduration=3.044359833 podStartE2EDuration="3.044359833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:31.043756957 +0000 UTC m=+1510.662416124" watchObservedRunningTime="2026-10-31 13:28:31.044359833 +0000 UTC m=+1510.663018995"
Oct 31 13:28:31.112314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:31.102588    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" podStartSLOduration=3.102568572 podStartE2EDuration="3.102568572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:31.100432228 +0000 UTC m=+1510.719091404" watchObservedRunningTime="2026-10-31 13:28:31.102568572 +0000 UTC m=+1510.721227736"
Oct 31 13:28:31.390236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:31.390206    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:31.391998 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:31.391970    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:28:31.392122 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:31.392022    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j" podUID="6a36003d-cda6-447e-bb9f-95f13e32c276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 13:28:31.780384 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:31.779460    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:31.791445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:31.787527    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:28:31.791445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:31.787677    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 13:28:31.832069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:31.832026    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:31.832487 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:31.832464    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:31.832689 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:31.832664    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj" podUID="11d0989b-0ce8-4041-bf02-e9af6b10c3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 13:28:32.393711 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:32.393674    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:28:32.394007 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:32.393977    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j" podUID="6a36003d-cda6-447e-bb9f-95f13e32c276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 13:28:32.778806 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:32.778716    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:28:32.779050 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:32.779022    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 13:28:32.826318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:32.826219    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:32.826318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:32.826278    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj" podUID="11d0989b-0ce8-4041-bf02-e9af6b10c3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 13:28:33.390103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:33.390068    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:28:33.390426 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:33.390383    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j" podUID="6a36003d-cda6-447e-bb9f-95f13e32c276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 13:28:33.781771 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:33.781671    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:28:33.781771 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:33.781750    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 13:28:33.833480 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:33.833444    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:33.834103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:33.834049    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj" podUID="11d0989b-0ce8-4041-bf02-e9af6b10c3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 13:28:34.390728 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:34.390689    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:28:34.391010 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:34.390978    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j" podUID="6a36003d-cda6-447e-bb9f-95f13e32c276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 13:28:34.780372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:34.779633    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:28:34.780372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:34.779694    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 13:28:34.832035 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:34.832000    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:34.832276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:34.832250    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj" podUID="11d0989b-0ce8-4041-bf02-e9af6b10c3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 13:28:35.394734 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:35.394696    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:28:35.395366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:35.395332    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j" podUID="6a36003d-cda6-447e-bb9f-95f13e32c276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 13:28:35.779912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:35.779206    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:28:35.779912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:35.779263    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 13:28:35.833815 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:35.833782    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:35.834537 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:35.834490    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj" podUID="11d0989b-0ce8-4041-bf02-e9af6b10c3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 13:28:36.391370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:36.391073    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:28:36.391370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:36.391134    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j" podUID="6a36003d-cda6-447e-bb9f-95f13e32c276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 13:28:36.500318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:36.492484    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:28:36.779376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:36.778591    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:36.781318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:36.779908    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:28:36.781318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:36.779957    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 13:28:36.827919 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:36.827885    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:36.828166 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:36.828140    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj" podUID="11d0989b-0ce8-4041-bf02-e9af6b10c3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 13:28:37.401318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:37.389985    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:28:37.401318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:37.390046    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j" podUID="6a36003d-cda6-447e-bb9f-95f13e32c276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 13:28:37.416333 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:37.414475    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:28:37.439128 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:37.439087    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:37.501450 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:37.501401    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:37.502733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:37.502709    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:28:37.802379 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:37.799408    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:28:37.802379 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:37.799472    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 13:28:37.830482 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:37.830439    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:37.830779 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:37.830726    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj" podUID="11d0989b-0ce8-4041-bf02-e9af6b10c3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 13:28:37.989324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:37.988855    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:38.080980 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:38.025789    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:28:38.390039 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:38.389993    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:28:38.390238 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:38.390086    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j" podUID="6a36003d-cda6-447e-bb9f-95f13e32c276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 13:28:38.778125 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:38.778025    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:28:38.778125 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:38.778089    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 13:28:38.826918 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:38.826877    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:38.827128 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:38.826956    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj" podUID="11d0989b-0ce8-4041-bf02-e9af6b10c3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 13:28:39.809613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:39.809580    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:39.811593 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:39.811570    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:39.817078 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:39.817059    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 13:28:39.848120 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:39.848092    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:39.855647 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:39.855623    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:39.859416 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:39.859380    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:28:39.943523 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:39.943491    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:39.945601 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:39.945579    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:39.949586 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:39.949565    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 13:28:43.647516 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.647480    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"]
Oct 31 13:28:43.653046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.653022    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.667873 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.667847    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"]
Oct 31 13:28:43.737126 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.737079    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d7ee133-8731-4bc3-bfa0-157935a79a24-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.737126 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.737125    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.737440 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.737261    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.737440 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.737332    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.737440 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.737364    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.737440 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.737404    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.737675 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.737509    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.737675 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.737645    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.737762 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.737698    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wsmh\" (UniqueName: \"kubernetes.io/projected/1d7ee133-8731-4bc3-bfa0-157935a79a24-kube-api-access-5wsmh\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.832083 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.832047    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"]
Oct 31 13:28:43.837657 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.837630    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:43.838590 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.838551    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.838727 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.838601    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.838727 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.838648    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.838853 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.838725    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.838853 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.838842    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.838971 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.838867    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wsmh\" (UniqueName: \"kubernetes.io/projected/1d7ee133-8731-4bc3-bfa0-157935a79a24-kube-api-access-5wsmh\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.838971 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.838934    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d7ee133-8731-4bc3-bfa0-157935a79a24-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.838971 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.838960    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.839080 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.839059    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.839540 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.839503    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.839776 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.839754    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.840015 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.839997    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.840942 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.840917    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.841201 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.841174    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d7ee133-8731-4bc3-bfa0-157935a79a24-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.852233 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.851146    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"]
Oct 31 13:28:43.854983 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.854209    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.862024 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.861969    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.874615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.872923    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.877284 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.876901    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wsmh\" (UniqueName: \"kubernetes.io/projected/1d7ee133-8731-4bc3-bfa0-157935a79a24-kube-api-access-5wsmh\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.970651 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.970614    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:43.972893 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.972859    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:43.973047 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.972940    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5smzr\" (UniqueName: \"kubernetes.io/projected/f696d368-e075-482d-8e6f-da424076afa0-kube-api-access-5smzr\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:43.973047 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.972978    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f696d368-e075-482d-8e6f-da424076afa0-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:43.973047 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.973009    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f696d368-e075-482d-8e6f-da424076afa0-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:43.973209 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.973048    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:43.973209 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.973130    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:43.973209 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.973166    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:43.973209 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.973194    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f696d368-e075-482d-8e6f-da424076afa0-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:43.975672 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:43.973479    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.113464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.108795    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5smzr\" (UniqueName: \"kubernetes.io/projected/f696d368-e075-482d-8e6f-da424076afa0-kube-api-access-5smzr\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.113464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.108855    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f696d368-e075-482d-8e6f-da424076afa0-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.113464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.108911    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f696d368-e075-482d-8e6f-da424076afa0-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.113464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.108960    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.113464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.109011    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.113464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.109051    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.113464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.109074    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f696d368-e075-482d-8e6f-da424076afa0-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.113464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.109181    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.113464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.109756    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.113464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.110720    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f696d368-e075-482d-8e6f-da424076afa0-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.113464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.111320    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.113464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.111563    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.113464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.112334    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.113464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.112848    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.124545 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.124153    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.132630 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.132214    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5smzr\" (UniqueName: \"kubernetes.io/projected/f696d368-e075-482d-8e6f-da424076afa0-kube-api-access-5smzr\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.135151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.135105    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f696d368-e075-482d-8e6f-da424076afa0-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.136887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.136842    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f696d368-e075-482d-8e6f-da424076afa0-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.212825 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.212517    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:44.367390 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:28:44.367361    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7ee133_8731_4bc3_bfa0_157935a79a24.slice/crio-3c63080d065bb8a87c051531d989ca1c32c07b6f69b078a73474d8494ed00c5c WatchSource:0}: Error finding container 3c63080d065bb8a87c051531d989ca1c32c07b6f69b078a73474d8494ed00c5c: Status 404 returned error can't find the container with id 3c63080d065bb8a87c051531d989ca1c32c07b6f69b078a73474d8494ed00c5c
Oct 31 13:28:44.368396 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.368323    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"]
Oct 31 13:28:44.370433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.370402    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:28:44.370526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.370463    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:28:44.370526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.370492    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:28:44.532346 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:28:44.532305    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf696d368_e075_482d_8e6f_da424076afa0.slice/crio-7abcf567ec8d4667468d150706d2ca72cdcc0e191a848b97d159d7c831575b5a WatchSource:0}: Error finding container 7abcf567ec8d4667468d150706d2ca72cdcc0e191a848b97d159d7c831575b5a: Status 404 returned error can't find the container with id 7abcf567ec8d4667468d150706d2ca72cdcc0e191a848b97d159d7c831575b5a
Oct 31 13:28:44.549931 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.549904    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"]
Oct 31 13:28:44.557691 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.557643    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:28:44.557900 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.557740    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:28:44.557900 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:44.557796    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:28:45.367671 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:45.367638    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c" event={"ID":"1d7ee133-8731-4bc3-bfa0-157935a79a24","Type":"ContainerStarted","Data":"f8136024f75f1426265ee27d8e5c5915b601b975a93d84f09330a30a939a1021"}
Oct 31 13:28:45.368051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:45.367693    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c" event={"ID":"1d7ee133-8731-4bc3-bfa0-157935a79a24","Type":"ContainerStarted","Data":"3c63080d065bb8a87c051531d989ca1c32c07b6f69b078a73474d8494ed00c5c"}
Oct 31 13:28:45.370882 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:45.370853    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq" event={"ID":"f696d368-e075-482d-8e6f-da424076afa0","Type":"ContainerStarted","Data":"2b07b498d9efd10c0642cb2af5a8def0d0aad2bb6ea163e27931efd44f4fdb66"}
Oct 31 13:28:45.371006 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:45.370898    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq" event={"ID":"f696d368-e075-482d-8e6f-da424076afa0","Type":"ContainerStarted","Data":"7abcf567ec8d4667468d150706d2ca72cdcc0e191a848b97d159d7c831575b5a"}
Oct 31 13:28:45.394560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:45.394511    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c" podStartSLOduration=2.394497477 podStartE2EDuration="2.394497477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:45.39396619 +0000 UTC m=+1525.012625380" watchObservedRunningTime="2026-10-31 13:28:45.394497477 +0000 UTC m=+1525.013156641"
Oct 31 13:28:45.456319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:45.446484    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq" podStartSLOduration=2.446459585 podStartE2EDuration="2.446459585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:45.429785866 +0000 UTC m=+1525.048445030" watchObservedRunningTime="2026-10-31 13:28:45.446459585 +0000 UTC m=+1525.065118750"
Oct 31 13:28:45.977828 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:45.977786    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:45.979182 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:45.979151    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 13:28:45.979314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:45.979215    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c" podUID="1d7ee133-8731-4bc3-bfa0-157935a79a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 13:28:46.217008 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:46.216978    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:46.217369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:46.217344    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 13:28:46.218683 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:46.218651    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq" podUID="f696d368-e075-482d-8e6f-da424076afa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 13:28:46.978158 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:46.978118    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 13:28:46.978584 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:46.978181    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c" podUID="1d7ee133-8731-4bc3-bfa0-157935a79a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 13:28:47.216540 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:47.214648    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 13:28:47.216540 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:47.214742    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq" podUID="f696d368-e075-482d-8e6f-da424076afa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 13:28:47.977790 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:47.977755    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 13:28:47.978040 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:47.978017    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c" podUID="1d7ee133-8731-4bc3-bfa0-157935a79a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 13:28:48.214314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:48.213724    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 13:28:48.214314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:48.213820    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq" podUID="f696d368-e075-482d-8e6f-da424076afa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 13:28:48.979319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:48.978732    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 13:28:48.979319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:48.978793    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c" podUID="1d7ee133-8731-4bc3-bfa0-157935a79a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 13:28:49.224369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.215632    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 13:28:49.224369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.215689    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq" podUID="f696d368-e075-482d-8e6f-da424076afa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 13:28:49.495315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.495217    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"]
Oct 31 13:28:49.538315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.533333    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.550350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.541567    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d85d82\"/\"default-dockercfg-jzcf5\""
Oct 31 13:28:49.550350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.543255    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d85d82\"/\"distinct-2-w1-d85d82-kserve-self-signed-certs\""
Oct 31 13:28:49.550350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.543409    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d85d82\"/\"kube-root-ca.crt\""
Oct 31 13:28:49.550350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.543562    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d85d82\"/\"openshift-service-ca.crt\""
Oct 31 13:28:49.613244 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.612343    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"]
Oct 31 13:28:49.622051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.621074    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-model-cache\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.622051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.621139    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-dshm\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.622051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.621215    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-home\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.622051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.621304    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbjrt\" (UniqueName: \"kubernetes.io/projected/d4bb048a-1211-4eea-b483-f1c22a9da6a5-kube-api-access-kbjrt\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.622051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.621346    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-tmp-dir\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.622051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.621513    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4bb048a-1211-4eea-b483-f1c22a9da6a5-tls-certs\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.723166 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.723128    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4bb048a-1211-4eea-b483-f1c22a9da6a5-tls-certs\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.730792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.730750    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-model-cache\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.731047 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.731024    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-dshm\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.731213 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.731199    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-home\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.731401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.731383    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbjrt\" (UniqueName: \"kubernetes.io/projected/d4bb048a-1211-4eea-b483-f1c22a9da6a5-kube-api-access-kbjrt\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.731536 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.731522    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-tmp-dir\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.732152 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.732130    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-tmp-dir\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.736673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.736641    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-home\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.736932 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.736907    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-model-cache\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.789539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.789465    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4bb048a-1211-4eea-b483-f1c22a9da6a5-tls-certs\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.790404 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.790379    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-dshm\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:49.876178 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:49.876128    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbjrt\" (UniqueName: \"kubernetes.io/projected/d4bb048a-1211-4eea-b483-f1c22a9da6a5-kube-api-access-kbjrt\") pod \"distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:50.095261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:50.095147    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 13:28:50.097552 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:50.097499    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c" podUID="1d7ee133-8731-4bc3-bfa0-157935a79a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 13:28:50.202318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:50.196019    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:50.240477 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:50.240033    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 13:28:50.240477 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:50.240093    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq" podUID="f696d368-e075-482d-8e6f-da424076afa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 13:28:50.715327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:50.714329    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"]
Oct 31 13:28:50.785289 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:28:50.785247    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:28:50.978137 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:50.978098    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 13:28:50.978412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:50.978165    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c" podUID="1d7ee133-8731-4bc3-bfa0-157935a79a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 13:28:51.213120 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:51.213080    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 13:28:51.213357 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:51.213149    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq" podUID="f696d368-e075-482d-8e6f-da424076afa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 13:28:51.698315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:51.697411    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" event={"ID":"d4bb048a-1211-4eea-b483-f1c22a9da6a5","Type":"ContainerStarted","Data":"7eef1bf5bae5166cbb033c5bfda43dea92d779cbd0f7c7ef4770776fc071ede5"}
Oct 31 13:28:51.698315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:51.697458    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" event={"ID":"d4bb048a-1211-4eea-b483-f1c22a9da6a5","Type":"ContainerStarted","Data":"a5a4e5dee28d3448d992d74ae5f29ad05db9eaef79d17041e158737fb67230ef"}
Oct 31 13:28:51.986384 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:51.986248    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:51.987243 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:51.987178    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:51.988626 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:51.988527    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:28:52.022675 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:52.017730    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podStartSLOduration=3.017710568 podStartE2EDuration="3.017710568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:51.736111735 +0000 UTC m=+1531.354770900" watchObservedRunningTime="2026-10-31 13:28:52.017710568 +0000 UTC m=+1531.636369735"
Oct 31 13:28:52.197187 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:52.197140    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:28:52.503597 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:52.503566    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:52.504574 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:52.504537    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:52.505496 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:52.505474    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 13:28:59.092318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:59.088391    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:59.202316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:59.197824    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:59.249317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:59.248448    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:28:59.412319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:59.411508    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:59.770370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:28:59.769443    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:00.036347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:00.032458    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:00.089450 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:00.089394    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:29:00.170267 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:00.170224    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded"
Oct 31 13:29:00.203319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:00.180373    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:00.203319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:00.200385    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:00.203691 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:00.203666    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:29:00.331626 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:00.331509    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:00.332986 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:00.332961    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:29:00.750436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:00.748496    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:29:00.776398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:00.776342    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:29:00.897317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:00.866866    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:29:00.897724 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:00.897700    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:29:01.010459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.010369    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:29:01.076320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.051524    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:29:01.078689 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.078651    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:01.159319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.095081    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:29:01.159319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.104328    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:29:01.159319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.127171    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:29:01.187074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.185180    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:01.187676 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.187654    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:29:01.448315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.445630    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded"
Oct 31 13:29:01.448315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.445767    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:29:01.512069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.512041    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:29:01.663901 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.663869    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:29:01.764363 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.763375    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:29:01.779317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.765237    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:29:01.891482 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.885378    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:01.929664 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.929624    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded"
Oct 31 13:29:02.000329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:01.996721    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:02.011685 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:02.010522    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:02.114819 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:02.114730    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:02.252319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:02.251402    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:02.288741 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:02.253097    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:29:02.453423 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:02.447491    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:02.453423 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:02.448061    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:29:02.719370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:02.716398    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:02.787317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:02.782620    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:29:02.829635 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:02.829600    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:29:02.894075 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:02.891411    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:02.997821 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:02.997635    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:29:03.017028 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:03.016855    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:03.026328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:03.019743    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:29:03.160953 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:03.160925    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:29:03.162543 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:03.162484    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:29:03.176671 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:03.170663    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:29:03.187552 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:03.183852    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:29:13.604935 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.604902    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"]
Oct 31 13:29:13.620663 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.620638    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"]
Oct 31 13:29:13.620968 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.620777    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.633611 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.633587    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"]
Oct 31 13:29:13.633750 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.633633    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"]
Oct 31 13:29:13.633750 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.633713    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.707999 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.707961    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16adae6b-a455-42f5-a51b-f9c018eec1cc-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.708343 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.708318    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.708544 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.708525    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.708705 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.708690    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.708867 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.708853    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.709024 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.709000    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.709166 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.709037    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b0354dc-4369-44e8-a995-5018598ea92a-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.709166 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.709090    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.709166 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.709150    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.709389 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.709177    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.709389 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.709242    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf6js\" (UniqueName: \"kubernetes.io/projected/7b0354dc-4369-44e8-a995-5018598ea92a-kube-api-access-rf6js\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.709580 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.709554    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b0354dc-4369-44e8-a995-5018598ea92a-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.709654 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.709621    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.709706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.709661    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.709773 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.709734    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.709839 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.709774    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.709839 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.709815    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b0354dc-4369-44e8-a995-5018598ea92a-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.709941 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.709867    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqnxz\" (UniqueName: \"kubernetes.io/projected/16adae6b-a455-42f5-a51b-f9c018eec1cc-kube-api-access-xqnxz\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.810833 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.810796    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqnxz\" (UniqueName: \"kubernetes.io/projected/16adae6b-a455-42f5-a51b-f9c018eec1cc-kube-api-access-xqnxz\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.811022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.810868    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16adae6b-a455-42f5-a51b-f9c018eec1cc-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.811022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.810903    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.811022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.810923    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.811022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.810941    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.811022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.810991    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.811318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.811026    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.811318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.811054    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b0354dc-4369-44e8-a995-5018598ea92a-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.811318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.811104    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.811318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.811136    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.811318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.811160    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.811318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.811224    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf6js\" (UniqueName: \"kubernetes.io/projected/7b0354dc-4369-44e8-a995-5018598ea92a-kube-api-access-rf6js\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.811318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.811275    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b0354dc-4369-44e8-a995-5018598ea92a-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.811687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.811341    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.811687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.811378    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.811687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.811465    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.811687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.811495    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.811687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.811532    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b0354dc-4369-44e8-a995-5018598ea92a-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.811998 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.811688    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.811998 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.811797    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16adae6b-a455-42f5-a51b-f9c018eec1cc-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.812118 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.812060    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.812118 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.812092    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.812364 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.812340    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b0354dc-4369-44e8-a995-5018598ea92a-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.812614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.812587    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.812706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.812604    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.812905 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.812879    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.813013 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.812984    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.813135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.813090    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.813702 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.813671    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.814092 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.814064    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.814263 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.814240    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b0354dc-4369-44e8-a995-5018598ea92a-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.815169 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.815146    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.823641 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.823615    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf6js\" (UniqueName: \"kubernetes.io/projected/7b0354dc-4369-44e8-a995-5018598ea92a-kube-api-access-rf6js\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.823819 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.823802    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqnxz\" (UniqueName: \"kubernetes.io/projected/16adae6b-a455-42f5-a51b-f9c018eec1cc-kube-api-access-xqnxz\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.824051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.824030    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b0354dc-4369-44e8-a995-5018598ea92a-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.824666 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.824647    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.903431 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.903400    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"]
Oct 31 13:29:13.909873 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.909851    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:13.920779 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.920757    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"]
Oct 31 13:29:13.928105 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.928081    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"]
Oct 31 13:29:13.933112 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.933086    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:13.946675 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.945585    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:13.956275 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.956250    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"]
Oct 31 13:29:13.963996 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.962928    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:13.993046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:13.993019    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"]
Oct 31 13:29:14.000119 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.000072    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.018340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.014061    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"]
Oct 31 13:29:14.018564 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.014066    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.018692 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.018675    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.018814 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.018802    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.018911 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.018898    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd64e87-c152-497f-9460-075f624e4157-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.019016 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.019005    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.019109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.019097    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.019216 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.019203    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.019328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.019315    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.019422 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.019410    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zmbb\" (UniqueName: \"kubernetes.io/projected/63fba834-3ec6-4542-97b5-5ddac5d4d450-kube-api-access-4zmbb\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.019512 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.019500    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzqgd\" (UniqueName: \"kubernetes.io/projected/8dd64e87-c152-497f-9460-075f624e4157-kube-api-access-hzqgd\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.019647 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.019636    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.019735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.019724    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fba834-3ec6-4542-97b5-5ddac5d4d450-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.019810 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.019799    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd64e87-c152-497f-9460-075f624e4157-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.019950 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.019939    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.020048 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.020037    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.020145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.020133    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.020231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.020220    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd64e87-c152-497f-9460-075f624e4157-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.020377 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.020362    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.048676 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.048505    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"]
Oct 31 13:29:14.054569 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.054544    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.072206 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.072165    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"]
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.125046    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.125081    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzqgd\" (UniqueName: \"kubernetes.io/projected/8dd64e87-c152-497f-9460-075f624e4157-kube-api-access-hzqgd\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.125109    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.125133    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.125154    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a7105e-a896-47cd-b203-feab72f829c3-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.125860    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.125948    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a7105e-a896-47cd-b203-feab72f829c3-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.125982    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126020    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126044    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126073    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126095    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd64e87-c152-497f-9460-075f624e4157-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126135    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126178    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126209    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm25x\" (UniqueName: \"kubernetes.io/projected/96a7105e-a896-47cd-b203-feab72f829c3-kube-api-access-bm25x\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126237    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126260    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126354    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126408    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126437    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126464    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd64e87-c152-497f-9460-075f624e4157-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126503    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126547    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126571    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126607    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126648    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zmbb\" (UniqueName: \"kubernetes.io/projected/63fba834-3ec6-4542-97b5-5ddac5d4d450-kube-api-access-4zmbb\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126697    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fba834-3ec6-4542-97b5-5ddac5d4d450-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126720    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd64e87-c152-497f-9460-075f624e4157-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126750    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126829    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjxcr\" (UniqueName: \"kubernetes.io/projected/5833e6de-e1f1-425c-865f-9293c4396f1e-kube-api-access-hjxcr\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126858    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126898    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126960    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.126984    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a7105e-a896-47cd-b203-feab72f829c3-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.127066    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.127105    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5833e6de-e1f1-425c-865f-9293c4396f1e-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.127134    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.127578    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.129436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.127835    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.131623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.130271    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.136984 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.132076    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.136984 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.135614    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.136984 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.135832    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fba834-3ec6-4542-97b5-5ddac5d4d450-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.136984 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.136271    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd64e87-c152-497f-9460-075f624e4157-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.136984 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.136553    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.136984 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.136811    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.137483 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.137001    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.141722 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.140421    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.141722 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.140831    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd64e87-c152-497f-9460-075f624e4157-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.141722 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.141218    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.144859 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.144835    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzqgd\" (UniqueName: \"kubernetes.io/projected/8dd64e87-c152-497f-9460-075f624e4157-kube-api-access-hzqgd\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.150153 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.148962    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd64e87-c152-497f-9460-075f624e4157-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.150153 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.150110    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zmbb\" (UniqueName: \"kubernetes.io/projected/63fba834-3ec6-4542-97b5-5ddac5d4d450-kube-api-access-4zmbb\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.151519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.151496    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.228713 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.228621    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a7105e-a896-47cd-b203-feab72f829c3-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.228713 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.228695    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.228926 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.228727    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5833e6de-e1f1-425c-865f-9293c4396f1e-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.228926 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.228784    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.228926 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.228811    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.228926 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.228839    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a7105e-a896-47cd-b203-feab72f829c3-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.228926 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.228873    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a7105e-a896-47cd-b203-feab72f829c3-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.228926 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.228901    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.229231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.228939    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.229231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.229000    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.229231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.229030    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm25x\" (UniqueName: \"kubernetes.io/projected/96a7105e-a896-47cd-b203-feab72f829c3-kube-api-access-bm25x\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.229231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.229059    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.229231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.229085    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.229231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.229112    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.229231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.229156    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.229231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.229161    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.229644 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.229394    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:14.229644 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.229540    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.229762 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.229644    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjxcr\" (UniqueName: \"kubernetes.io/projected/5833e6de-e1f1-425c-865f-9293c4396f1e-kube-api-access-hjxcr\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.229762 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.229672    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.229762 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.229700    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5833e6de-e1f1-425c-865f-9293c4396f1e-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.229936 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.229913    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a7105e-a896-47cd-b203-feab72f829c3-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.230153 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.230115    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.230255 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.230169    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.230378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.230348    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.230378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.230358    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.230843 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.230706    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.230843 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.230730    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.231038 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.230890    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.232522 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.232495    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.233543 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.233516    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.233543 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.233519    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.233700 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.233655    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a7105e-a896-47cd-b203-feab72f829c3-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.241433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.241408    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a7105e-a896-47cd-b203-feab72f829c3-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.241601 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.241577    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm25x\" (UniqueName: \"kubernetes.io/projected/96a7105e-a896-47cd-b203-feab72f829c3-kube-api-access-bm25x\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.241905 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.241850    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.242283 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.242262    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjxcr\" (UniqueName: \"kubernetes.io/projected/5833e6de-e1f1-425c-865f-9293c4396f1e-kube-api-access-hjxcr\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.271766 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.271570    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:14.316028 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.315999    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"]
Oct 31 13:29:14.334656 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.334077    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.334656 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.334142    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.334656 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.334171    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.337372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.337346    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"]
Oct 31 13:29:14.341276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.340790    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.341276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.340865    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.341276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.340903    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.358761 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.358560    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:14.401198 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.400915    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:14.651381 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.651165    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.668416 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.668247    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.668416 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.668392    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.711398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.711360    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.711631 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.711612    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.714909 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.714880    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.728258 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.728228    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"]
Oct 31 13:29:14.728383 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.728322    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"]
Oct 31 13:29:14.728383 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.728356    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"]
Oct 31 13:29:14.733178 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:29:14.733124    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5833e6de_e1f1_425c_865f_9293c4396f1e.slice/crio-f2be43283708bc3ef2174d333a22562a0eb9b68d1fd8b29b79fece688106f6df WatchSource:0}: Error finding container f2be43283708bc3ef2174d333a22562a0eb9b68d1fd8b29b79fece688106f6df: Status 404 returned error can't find the container with id f2be43283708bc3ef2174d333a22562a0eb9b68d1fd8b29b79fece688106f6df
Oct 31 13:29:14.741280 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.740534    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.741280 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.740622    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.741280 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.740667    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.957162 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.957135    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"]
Oct 31 13:29:14.966201 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:29:14.966143    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a7105e_a896_47cd_b203_feab72f829c3.slice/crio-b5c6b6ccc13aadff9429ee599e0eddf6fff6349f2925eabf8017d9b8375deddb WatchSource:0}: Error finding container b5c6b6ccc13aadff9429ee599e0eddf6fff6349f2925eabf8017d9b8375deddb: Status 404 returned error can't find the container with id b5c6b6ccc13aadff9429ee599e0eddf6fff6349f2925eabf8017d9b8375deddb
Oct 31 13:29:14.971085 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.969710    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.971085 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.969804    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:14.971085 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:14.969890    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:15.471318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.456389    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" event={"ID":"96a7105e-a896-47cd-b203-feab72f829c3","Type":"ContainerStarted","Data":"1dbb076eaceb11cedc9d12913ee07374dddbb8926f27fa60586c47e11014253f"}
Oct 31 13:29:15.471318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.456458    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" event={"ID":"96a7105e-a896-47cd-b203-feab72f829c3","Type":"ContainerStarted","Data":"b5c6b6ccc13aadff9429ee599e0eddf6fff6349f2925eabf8017d9b8375deddb"}
Oct 31 13:29:15.476412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.476237    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" event={"ID":"63fba834-3ec6-4542-97b5-5ddac5d4d450","Type":"ContainerStarted","Data":"3a3aa6fd747210a7a1571b2ee7ddf5c29e9d576c9ddea2ff14a5fd0e0ce22ab1"}
Oct 31 13:29:15.476658 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.476634    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" event={"ID":"63fba834-3ec6-4542-97b5-5ddac5d4d450","Type":"ContainerStarted","Data":"b903f928d256c49d60ebaf5957934e258e110ec3b6736b6d483b413350d8e7a2"}
Oct 31 13:29:15.485207 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.485178    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" event={"ID":"8dd64e87-c152-497f-9460-075f624e4157","Type":"ContainerStarted","Data":"235be30bad4a3ebbd8ed56f635ac3a1b3ccaf8ff26f881a64437ee94c0c7323d"}
Oct 31 13:29:15.485703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.485671    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" event={"ID":"8dd64e87-c152-497f-9460-075f624e4157","Type":"ContainerStarted","Data":"1477405b422e109c078580db11193539635ebd98bbd6e2133c1694ffb3f692eb"}
Oct 31 13:29:15.486139 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.486120    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" event={"ID":"7b0354dc-4369-44e8-a995-5018598ea92a","Type":"ContainerStarted","Data":"c4d7f140762fc842a63c223b2af56c155d1157c76436a62771308a265bc1288f"}
Oct 31 13:29:15.490632 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.490612    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" event={"ID":"7b0354dc-4369-44e8-a995-5018598ea92a","Type":"ContainerStarted","Data":"41bb2b1fb56bf50b918bc74b37f6ce090f628ad42804e89867a3489a40afab3b"}
Oct 31 13:29:15.490799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.490781    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" event={"ID":"16adae6b-a455-42f5-a51b-f9c018eec1cc","Type":"ContainerStarted","Data":"125ded9fb17944d870b98a5607223b07032c8ac8bceb93b3bd11f11a48703e57"}
Oct 31 13:29:15.491014 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.490998    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" event={"ID":"16adae6b-a455-42f5-a51b-f9c018eec1cc","Type":"ContainerStarted","Data":"15e316b00b2b1a965c2b3f192e2065d5ba77cbe335c9f42d51b8e195237e5e78"}
Oct 31 13:29:15.491255 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.491236    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" event={"ID":"5833e6de-e1f1-425c-865f-9293c4396f1e","Type":"ContainerStarted","Data":"1deba25e69ae1f08e3994501aa5bc4ba375416444e71749d9fba2f7e93733caf"}
Oct 31 13:29:15.491428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.491413    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" event={"ID":"5833e6de-e1f1-425c-865f-9293c4396f1e","Type":"ContainerStarted","Data":"f2be43283708bc3ef2174d333a22562a0eb9b68d1fd8b29b79fece688106f6df"}
Oct 31 13:29:15.639462 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.639362    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podStartSLOduration=1.6393368289999999 podStartE2EDuration="1.639336829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:15.579162271 +0000 UTC m=+1555.197821436" watchObservedRunningTime="2026-10-31 13:29:15.639336829 +0000 UTC m=+1555.257995994"
Oct 31 13:29:15.640446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.640393    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podStartSLOduration=2.640373291 podStartE2EDuration="2.640373291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:15.618748438 +0000 UTC m=+1555.237407639" watchObservedRunningTime="2026-10-31 13:29:15.640373291 +0000 UTC m=+1555.259032455"
Oct 31 13:29:15.710110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.709943    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podStartSLOduration=2.709916723 podStartE2EDuration="2.709916723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:15.70563692 +0000 UTC m=+1555.324296084" watchObservedRunningTime="2026-10-31 13:29:15.709916723 +0000 UTC m=+1555.328575950"
Oct 31 13:29:15.912683 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.912614    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" podStartSLOduration=2.912593241 podStartE2EDuration="2.912593241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:15.786848472 +0000 UTC m=+1555.405507638" watchObservedRunningTime="2026-10-31 13:29:15.912593241 +0000 UTC m=+1555.531252407"
Oct 31 13:29:16.003686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.960786    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" podStartSLOduration=2.96076412 podStartE2EDuration="2.96076412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:15.81462745 +0000 UTC m=+1555.433286614" watchObservedRunningTime="2026-10-31 13:29:15.96076412 +0000 UTC m=+1555.579423284"
Oct 31 13:29:16.003686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.971929    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:16.003686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.982791    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:29:16.003686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.982844    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 13:29:16.003686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.984955    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:16.003686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.985066    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 13:29:16.003686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:15.985108    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 13:29:16.012312 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.008955    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podStartSLOduration=3.008935134 podStartE2EDuration="3.008935134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:15.969731879 +0000 UTC m=+1555.588391036" watchObservedRunningTime="2026-10-31 13:29:16.008935134 +0000 UTC m=+1555.627594300"
Oct 31 13:29:16.232486 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.232341    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:16.284282 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.262409    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:16.284282 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.262474    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 13:29:16.284282 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.281157    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:16.284282 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.281332    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:16.284282 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.281379    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 13:29:16.408316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.379376    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:16.408316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.391889    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:16.408316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.391955    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 13:29:16.408316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.405378    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:16.438484 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.438445    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:16.438746 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.438706    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 13:29:16.680318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.679502    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:16.947775 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.947668    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:29:16.948348 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.948318    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 13:29:16.976322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.971448    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 13:29:16.976322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.971521    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 13:29:16.976322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:16.971615    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:17.006259 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.006200    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:17.118546 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.118495    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:17.176318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.175395    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:17.193317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.176528    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:17.232899 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.232371    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:17.232899 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.232442    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 13:29:17.263319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.254396    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:17.273999 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.273395    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:17.288492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.287883    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:17.288492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.287948    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 13:29:17.362159 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.360773    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:17.362159 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.360830    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 13:29:17.417473 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.409901    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:17.417473 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.409968    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 13:29:17.738327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.727497    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:29:17.738327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.728672    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:29:17.825567 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.825527    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:29:17.996052 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:17.995960    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:18.157190 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.150802    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:18.159443 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.050816    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 13:29:18.159630 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.159601    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 13:29:18.175902 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.175862    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:18.182170 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.132608    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:29:18.183828 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.183803    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:29:18.184064 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.184048    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:29:18.184233 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.184221    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:29:18.200319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.192693    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:29:18.200319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.192751    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 13:29:18.302398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.271684    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:18.302398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.271750    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 13:29:18.302398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.272733    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:18.302398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.272788    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 13:29:18.302398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.276227    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:29:18.302398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.295452    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:29:18.359320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.358429    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:18.359320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.358898    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:29:18.404541 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.404505    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:18.404783 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.404754    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 13:29:18.460537 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.460500    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:18.565370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.560185    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 13:29:18.565370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.552625    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:29:18.915525 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.915479    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:18.943031 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.942989    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:29:18.969586 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.969550    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:29:18.969848 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:18.969822    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 13:29:19.133339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.083458    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 13:29:19.133339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.083522    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 13:29:19.347395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.335629    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:19.347707 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.347679    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:19.347884 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.347858    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 13:29:19.400346 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.382513    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:19.400346 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.395433    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:19.404434 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.404394    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:19.404683 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.404661    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:19.431410 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.408693    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:19.439348 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.437774    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:19.478369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.437837    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:19.503389 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.439747    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:29:19.507507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.507485    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:29:19.657410 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.503713    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:19.765328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.503789    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:19.765328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.757453    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 13:29:19.765328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.681112    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:29:19.790790 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.503615    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 13:29:19.831752 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.543464    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:19.832218 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:19.832014    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 13:29:20.058233 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.011432    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:29:20.058233 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.011502    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 13:29:20.058233 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.035922    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 13:29:20.058233 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.036034    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 13:29:20.074193 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.066521    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:29:20.107317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.082837    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:20.225316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.190813    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:20.352878 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.274508    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:20.353874 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.353842    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 13:29:20.354086 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.352498    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:20.355370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.327281    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:20.355531 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.355511    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 13:29:20.379769 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.379728    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:29:20.383648 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.383615    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:20.410540 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.396912    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:20.410540 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.396974    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 13:29:20.410540 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.399386    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:20.427550 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.425386    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:20.470333 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.466016    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:20.470333 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.466072    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 13:29:20.488464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.488420    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:29:20.490038 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.490012    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:29:20.816318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.794435    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:20.867435 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.867395    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:20.868479 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.868442    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:29:20.991150 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.991096    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:29:20.991426 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:20.991398    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 13:29:21.099371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.065448    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:21.099371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.090796    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 13:29:21.099371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.090844    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 13:29:21.099371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.091801    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:21.242319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.235153    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:29:21.367042 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.245376    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:21.496406 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.317424    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:21.496668 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.496635    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 13:29:21.496780 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.416575    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:29:21.497145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.356503    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:21.504679 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.417228    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:29:21.512313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.428280    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:21.512313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.460902    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:21.512614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.512595    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:29:21.521730 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.417194    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded"
Oct 31 13:29:21.521935 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.317472    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:21.522059 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.522039    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 13:29:21.522155 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.425948    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:21.522269 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.522251    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 13:29:21.522439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.482422    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:21.528425 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.522537    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 13:29:21.530735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.494354    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:21.820475 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.820422    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:29:21.919317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.914186    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:21.919573 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.897276    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:21.996217 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.996180    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:29:21.996531 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:21.996501    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 13:29:22.007439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.007410    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 13:29:22.007692 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.007666    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 13:29:22.099369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.096025    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:22.282615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.278455    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:22.282615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.278512    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 13:29:22.282615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.278785    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:22.282615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.278817    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 13:29:22.328577 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:29:22.328527    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:29:22.406337 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.403399    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:22.429592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.429558    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:22.429857 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.429831    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 13:29:22.438808 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.438777    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:22.439121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.439090    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 13:29:22.508562 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.508520    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:22.509054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.508934    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:22.509241 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.509215    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:29:22.509614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.509590    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:22.509988 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.509238    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:29:22.533607 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.533572    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:29:22.743450 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.743354    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded"
Oct 31 13:29:22.832893 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.832527    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:22.920486 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.920438    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:29:22.955528 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.955491    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:29:22.955803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.955775    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 13:29:22.970841 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.970808    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 13:29:22.971114 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:22.971083    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 13:29:23.120380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.119854    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:29:23.311388 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.311355    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:29:23.361317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.334434    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:23.361317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.360455    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 13:29:23.372361 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.362665    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:23.372361 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.362712    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 13:29:23.377925 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.377532    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:23.377925 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.377582    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 13:29:23.410970 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.410922    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:23.515495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.488499    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:23.515495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.488561    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 13:29:23.534058 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.534017    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:23.622352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.563142    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:29:23.622352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.563543    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:29:23.622352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.575570    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:23.627360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.627253    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:23.629195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.629162    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:23.834315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.832806    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:23.944536 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:23.944434    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:29:24.091318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.087396    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:29:24.102218 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.102168    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 13:29:24.116397 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.092641    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 13:29:24.116663 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.116627    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 13:29:24.129909 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.129401    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:24.243376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.243238    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:24.243376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.243328    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 13:29:24.287262 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.276579    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:24.287262 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.276655    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 13:29:24.361330 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.360278    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:24.361330 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.360357    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 13:29:24.407480 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.407437    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:24.411213 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.408888    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:24.411213 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.408934    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 13:29:24.412313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.411590    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:24.547974 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.547877    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:29:24.581349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.565524    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:24.581959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.574376    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:24.588350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.578358    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:24.675321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.660465    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:24.775476 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.775443    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:29:24.846079 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:24.845981    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:29:25.047783 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.047744    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:25.108374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.100820    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 13:29:25.147417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.147371    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 13:29:25.197451 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.197403    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:25.290938 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.136515    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:29:25.380369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.342583    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 13:29:25.514317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.282858    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:25.514802 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.514764    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 13:29:25.540317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.414337    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:25.644369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.631865    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:29:25.644369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.283560    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:25.644369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.632996    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 13:29:25.644369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.583902    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:25.644369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.583860    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:25.644369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.558427    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:29:25.645862 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.583924    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:29:25.654402 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.381675    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:25.654402 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.652960    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 13:29:25.654402 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.432460    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:25.654402 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.653013    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 13:29:25.671200 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.670163    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:25.731319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.705980    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:29:25.896497 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.895310    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:26.007524 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:25.989214    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:29:26.108561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.108497    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:29:26.108881 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.108852    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 13:29:26.118078 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.109099    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 13:29:26.118389 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.118346    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 13:29:26.383375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.360365    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:26.469668 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.384360    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:26.469668 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.384438    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 13:29:26.469668 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.447817    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:26.469668 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.447872    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 13:29:26.495606 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.495569    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:26.556319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.534521    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 13:29:26.664374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.663547    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:29:26.695466 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.663708    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:29:26.741358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.706854    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:29:26.741358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.727334    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:26.741358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.727418    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 13:29:26.741358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.728277    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:29:26.817744 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.817701    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:26.926604 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.926486    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:26.998452 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.998413    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:29:27.041559 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.041529    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 13:29:27.067319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.041439    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:29:27.067865 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:26.998485    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 13:29:27.068054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.068026    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 13:29:27.107320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.029621    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:27.300744 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.300641    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:27.300997 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.300970    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 13:29:27.302023 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.301902    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:27.302203 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.302184    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 13:29:27.370097 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.370047    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:27.374157 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.374119    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:27.374390 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.374366    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 13:29:27.431895 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.431852    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:27.453565 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.453529    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 13:29:27.730320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.702468    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:27.740388 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.736383    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:27.740388 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.738426    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:27.741448 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.741398    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:27.823587 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.823528    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:27.900455 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.900409    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:29:27.994415 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.992436    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 13:29:27.994415 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.992497    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 13:29:27.997409 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:27.997370    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:28.071785 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.069397    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:29:28.071785 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.070154    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:29:28.086268 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.086238    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:29:28.242317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.236443    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:29:28.326465 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.304652    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:28.326465 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.304713    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 13:29:28.413341 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.400854    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:28.413620 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.413583    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 13:29:28.427320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.401803    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:28.427568 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.427536    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 13:29:28.434321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.427990    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:28.434321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.428037    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 13:29:28.527321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.500359    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:28.527321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.500731    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:29:28.742333 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.738871    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:28.748544 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.748474    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:28.764169 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.764125    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:28.825559 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.825516    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:28.903858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.903801    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:28.943312 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.940112    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:29:28.943312 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.940268    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:29:28.943312 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.940522    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"]
Oct 31 13:29:28.956406 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.956375    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:28.956660 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.956631    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:28.972790 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.966646    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 13:29:28.972790 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:28.966695    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 13:29:29.010643 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.010535    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:29.074639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.074586    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:29.227315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.201638    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:29.227701 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.227649    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:29.290982 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.289491    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:29.295008 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.293988    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:29.295008 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.294035    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 13:29:29.379409 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.379376    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:29.379655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.379630    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 13:29:29.405867 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.405834    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:29.406122 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.406093    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 13:29:29.459319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.455622    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:29.459319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.455693    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:29.459319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.455737    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:29.459319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.455846    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:29.459319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.455890    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:29.459319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.455923    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"]
Oct 31 13:29:29.459319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.455974    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:29.459319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.456040    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:29:29.459319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.456076    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:29:29.459319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.456134    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:29:29.459319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.456158    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:29.459319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.456223    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:29:29.461415 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.461376    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.483694 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.481595    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:29:29.503910 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.503888    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:29:29.622078 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.622044    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.622402 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.622383    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdwhr\" (UniqueName: \"kubernetes.io/projected/4419c5c1-3c78-4b4f-afdc-17714fe39c54-kube-api-access-wdwhr\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.699102 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.695422    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.699102 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.695495    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.699102 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.695630    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.699102 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.695811    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.699102 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.695880    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.699102 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.695944    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.699102 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.695985    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.812510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.799412    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.812510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.799491    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.812510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.799538    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.812510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.799572    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.812510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.799603    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.812510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.799672    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.812510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.799720    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdwhr\" (UniqueName: \"kubernetes.io/projected/4419c5c1-3c78-4b4f-afdc-17714fe39c54-kube-api-access-wdwhr\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.812510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.799881    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.812510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.799924    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.812510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.800402    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.812510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.801116    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.860004 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.826625    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:29:29.860004 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.841047    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.860004 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.859246    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.933450 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.904116    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.933450 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.909598    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:29.945347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:29.939915    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:30.062372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:30.012601    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:29:30.098582 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:30.097448    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:30.099183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:30.099157    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 13:29:30.100838 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:30.100808    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 13:29:30.103538 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:30.101199    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdwhr\" (UniqueName: \"kubernetes.io/projected/4419c5c1-3c78-4b4f-afdc-17714fe39c54-kube-api-access-wdwhr\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:30.140582 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:30.140548    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:30.193649 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:30.193535    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:30.332319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:30.331544    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:30.332319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:30.331611    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:30.359253 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:30.345707    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:30.359253 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:30.345775    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 13:29:30.691319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:30.688584    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:30.693015 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:30.692951    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:29:30.844798 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:30.830465    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:30.875448 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:30.853198    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:29:31.091637 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:31.091558    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:31.095836 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:31.095812    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:29:31.166746 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:31.166716    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:29:31.290445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:31.290394    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:31.290706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:31.290674    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 13:29:31.373044 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:31.348730    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:31.373044 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:31.372645    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:31.373044 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:31.372756    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:31.373044 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:31.372814    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:31.376230 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:31.376202    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:29:31.383222 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:31.383192    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"]
Oct 31 13:29:31.492411 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:31.492385    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:31.518365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:31.513128    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:29:31.889359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:31.880913    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" event={"ID":"4419c5c1-3c78-4b4f-afdc-17714fe39c54","Type":"ContainerStarted","Data":"4d0d37f91315a91f0c49dc8c41bf59688e519435d7adab84f52db038476665ba"}
Oct 31 13:29:31.889359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:31.880976    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" event={"ID":"4419c5c1-3c78-4b4f-afdc-17714fe39c54","Type":"ContainerStarted","Data":"c9415d4fd0548e963c75c1c3e406ed6b29bd8ee1bf2d83f1489518637d101022"}
Oct 31 13:29:31.963935 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:31.943284    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podStartSLOduration=3.9432622139999998 podStartE2EDuration="3.943262214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:31.936816041 +0000 UTC m=+1571.555475232" watchObservedRunningTime="2026-10-31 13:29:31.943262214 +0000 UTC m=+1571.561921377"
Oct 31 13:29:32.150704 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:32.150415    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:32.150704 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:32.150534    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:32.151260 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:32.150985    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 13:29:32.292517 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:32.292482    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:32.294784 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:32.294383    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:29:33.151694 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:33.151658    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:33.152262 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:33.152234    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 13:29:34.155716 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:34.155676    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:34.156328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:34.156274    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 13:29:34.222318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:34.209449    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:34.390318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:34.387455    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:34.390318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:34.388741    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:34.458692 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:34.458590    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:34.508439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:34.508389    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:34.509770 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:34.509731    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:34.531588 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:34.531541    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:34.603319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:34.599362    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:34.848497 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:34.848388    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:29:34.899082 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:34.859382    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:29:35.186317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.148727    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:35.186317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.148793    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 13:29:35.247338 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.238779    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:35.406151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.406115    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:29:35.456371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.407134    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:35.500348 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.458916    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:29:35.640323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.509546    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:35.640524 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.509588    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:35.640819 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.512649    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:29:35.640913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.640888    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:29:35.640975 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.640960    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:29:35.641027 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.641017    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:29:35.643536 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.643500    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:35.644946 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.644928    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:29:35.678329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.594418    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:29:35.678329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.678104    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:29:35.686427 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.513306    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:35.746413 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.730538    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:29:35.830985 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.830950    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:29:35.904317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.901176    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:29:35.924319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.907052    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:29:35.924319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.908224    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:29:35.988272 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:35.988238    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:29:36.216588 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:36.216544    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:36.227448 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:36.227402    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 13:29:36.398319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:36.391447    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:36.492518 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:36.492423    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:36.676319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:36.675532    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:36.676500 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:36.676349    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:29:36.708318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:36.700051    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:29:36.751372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:36.711394    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:29:36.799321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:36.752758    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:36.944370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:36.944327    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:36.999751 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:36.999710    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:29:37.289318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:37.267075    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:37.308248 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:37.308202    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 13:29:37.430538 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:37.405656    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:37.490861 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:37.490826    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:29:37.493695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:37.493655    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:37.710677 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:37.710629    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:37.712106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:37.712073    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:29:37.739317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:37.736866    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:37.780346 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:37.780198    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:37.780346 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:37.780259    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp" podUID="c290e457-707d-4d9a-89e0-208ccb07111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:37.850462 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:37.850406    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:37.851108 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:37.851076    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:37.981377 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:37.970431    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:38.000359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.000315    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:38.158942 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.158486    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:38.158942 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.158551    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 13:29:38.258507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.258402    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:38.258813 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.258771    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6" podUID="8906fc94-2d02-49d6-8ab0-c578e4b7391a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:38.276468 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.276436    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:38.276747 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.276716    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs" podUID="13e81951-495d-4169-bdbb-f6484e5d4756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:38.435527 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.435467    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:38.491323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.490279    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:38.509328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.507588    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:29:38.720319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.713401    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:38.720319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.713759    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:29:38.746784 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.746403    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:38.881376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.860459    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:29:38.881376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.860772    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:29:38.881376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.861036    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:29:38.971127 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.970409    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:38.971127 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:38.970721    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:39.079439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:39.079394    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:39.177406 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:39.115256    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:39.177406 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:39.159639    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:39.177406 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:39.159696    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 13:29:39.200762 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:39.200455    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:39.439493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:39.439390    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:29:39.508565 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:39.508514    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:39.534457 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:39.534391    4660 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:39.535013 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:39.534978    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg" podUID="5c132a2a-1e7e-4858-80fb-376083a5b0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:39.718535 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:39.718426    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:29:39.719121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:39.718759    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:39.747582 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:39.747533    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:29:39.883445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:39.883391    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:29:39.883933 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:39.883767    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:29:39.987565 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:39.987462    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:29:40.086319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:40.085398    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:29:40.114153 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:40.114110    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:40.157672 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:40.157636    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:40.157934 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:40.157903    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 13:29:40.453319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:40.442405    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:40.494923 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:40.494878    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:40.651319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:40.647403    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:29:40.739704 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:40.739593    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:40.740015 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:40.739983    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:29:40.769314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:40.765850    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:29:40.893675 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:40.892579    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:29:40.893675 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:40.892887    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:29:40.907324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:40.905373    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:40.907324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:40.905428    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj" podUID="11d0989b-0ce8-4041-bf02-e9af6b10c3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.011378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.005408    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:41.011378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.005478    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.039319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.018774    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:29:41.139634 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.099747    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.259804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.152110    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:41.280686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.260025    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 13:29:41.303419 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.152533    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.395318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.394187    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.474496 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.474448    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:29:41.540634 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.532377    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.541730 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.541706    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:29:41.670439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.670395    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.742472 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.742428    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.906588 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.742707    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.917323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.774692    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.918333 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.892905    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.929350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.894822    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.929350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:41.894852    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:29:42.019319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:42.019071    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:42.154505 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:42.154469    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:42.154770 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:42.154743    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 13:29:42.326829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:42.326737    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:42.489190 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:42.489150    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:29:42.577969 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:42.576850    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:42.610350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:42.610318    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:29:42.702433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:42.702384    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:42.716967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:42.716928    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:42.864369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:42.860809    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:29:42.899393 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:42.899343    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:29:42.977487 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:42.907177    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:42.977665 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:42.939400    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:43.092323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:43.090103    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:29:43.098312 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:43.095699    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:29:43.127367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:43.121617    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:29:43.147326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:43.146661    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:29:43.151693 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:43.151660    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:43.189313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:43.188901    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:43.189313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:43.188968    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 13:29:43.190958 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:43.190172    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:29:43.194094 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:43.194065    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:29:43.194544 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:43.194512    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:29:43.285339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:43.281411    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:29:44.150059 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:44.149682    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:44.150059 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:44.149743    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 13:29:45.149198 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:45.149161    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:45.149513 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:45.149478    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:45.149655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:45.149549    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 13:29:46.648176 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:46.648066    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:46.650141 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:46.650117    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:29:50.751990 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:29:50.751925    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:29:58.781826 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:58.781786    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"]
Oct 31 13:29:58.820070 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:58.817723    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"]
Oct 31 13:29:58.820070 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:58.817877    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:58.964115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:58.964087    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2zhh\" (UniqueName: \"kubernetes.io/projected/6acc3b99-906c-429c-99f8-3aee4333eb66-kube-api-access-f2zhh\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:58.964354 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:58.964148    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6acc3b99-906c-429c-99f8-3aee4333eb66-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:58.964354 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:58.964264    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:58.964504 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:58.964435    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:58.964504 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:58.964494    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:58.964605 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:58.964536    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:58.964695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:58.964678    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:58.964736 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:58.964714    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:58.964794 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:58.964749    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.065740 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.065638    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.065740 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.065694    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.065975 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.065739    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.065975 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.065817    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2zhh\" (UniqueName: \"kubernetes.io/projected/6acc3b99-906c-429c-99f8-3aee4333eb66-kube-api-access-f2zhh\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.065975 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.065879    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6acc3b99-906c-429c-99f8-3aee4333eb66-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.065975 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.065925    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.066187 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.066014    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.066187 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.066066    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.066187 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.066107    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.066396 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.066177    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.066396 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.066273    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.066514 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.066444    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.066596 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.066573    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.066670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.066650    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6acc3b99-906c-429c-99f8-3aee4333eb66-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.068255 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.068234    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.070279 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.070256    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.078673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.078613    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.078773 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.078688    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2zhh\" (UniqueName: \"kubernetes.io/projected/6acc3b99-906c-429c-99f8-3aee4333eb66-kube-api-access-f2zhh\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.151234 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.150926    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:29:59.154146 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.154116    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"]
Oct 31 13:29:59.162640 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.162619    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.177815 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.177099    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"]
Oct 31 13:29:59.229692 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.205096    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"]
Oct 31 13:29:59.233492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.232709    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"]
Oct 31 13:29:59.233492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.232821    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.272323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.269850    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.272323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.269923    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.272323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.270017    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.272323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.270158    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa19833b-88b6-453e-81c8-4ce97c30669e-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.272323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.270226    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.272323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.270383    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpcq9\" (UniqueName: \"kubernetes.io/projected/aa19833b-88b6-453e-81c8-4ce97c30669e-kube-api-access-hpcq9\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.272323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.270446    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.272323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.270478    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.272323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.270527    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.372382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.372348    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1daa31f8-4e70-413f-9529-0a67eef57732-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.372561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.372436    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.372631 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.372538    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.372975 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.372951    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.373052 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.373032    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.373111 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.372620    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.373111 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.373097    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.373218 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.373176    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1daa31f8-4e70-413f-9529-0a67eef57732-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.373218 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.373206    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1daa31f8-4e70-413f-9529-0a67eef57732-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.373379 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.373227    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa19833b-88b6-453e-81c8-4ce97c30669e-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.373379 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.373245    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.373379 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.373328    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.373379 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.373364    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.373592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.373384    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.373592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.373411    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.373592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.373453    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpcq9\" (UniqueName: \"kubernetes.io/projected/aa19833b-88b6-453e-81c8-4ce97c30669e-kube-api-access-hpcq9\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.373592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.373481    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4j2\" (UniqueName: \"kubernetes.io/projected/1daa31f8-4e70-413f-9529-0a67eef57732-kube-api-access-tn4j2\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.373592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.373512    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.373592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.373539    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.373929 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.373906    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.374194 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.374175    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa19833b-88b6-453e-81c8-4ce97c30669e-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.374355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.374330    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.375010 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.374988    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.377623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.377185    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.377623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.377603    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.387134 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.387109    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.387594 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.387575    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpcq9\" (UniqueName: \"kubernetes.io/projected/aa19833b-88b6-453e-81c8-4ce97c30669e-kube-api-access-hpcq9\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.475521 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.475487    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.475817 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.475530    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.475817 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.475578    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.475817 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.475643    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4j2\" (UniqueName: \"kubernetes.io/projected/1daa31f8-4e70-413f-9529-0a67eef57732-kube-api-access-tn4j2\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.475817 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.475795    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1daa31f8-4e70-413f-9529-0a67eef57732-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.476030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.475938    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.476030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.475951    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.476131 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.476048    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.476131 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.476077    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1daa31f8-4e70-413f-9529-0a67eef57732-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.476244 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.476133    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1daa31f8-4e70-413f-9529-0a67eef57732-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.476244 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.476137    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.476739 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.476575    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.476863 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.476738    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.476927 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.476883    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1daa31f8-4e70-413f-9529-0a67eef57732-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.479072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.479045    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.479420 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.479402    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1daa31f8-4e70-413f-9529-0a67eef57732-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.489792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.489287    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1daa31f8-4e70-413f-9529-0a67eef57732-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.489792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.489548    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4j2\" (UniqueName: \"kubernetes.io/projected/1daa31f8-4e70-413f-9529-0a67eef57732-kube-api-access-tn4j2\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.501947 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:29:59.501917    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6acc3b99_906c_429c_99f8_3aee4333eb66.slice/crio-073caf68985e2a7da2f9c3c6c7e1e78d0a4ad852a3605a8a753f0c13ea54ed7b WatchSource:0}: Error finding container 073caf68985e2a7da2f9c3c6c7e1e78d0a4ad852a3605a8a753f0c13ea54ed7b: Status 404 returned error can't find the container with id 073caf68985e2a7da2f9c3c6c7e1e78d0a4ad852a3605a8a753f0c13ea54ed7b
Oct 31 13:29:59.503347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.503284    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"]
Oct 31 13:29:59.505090 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.505051    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:59.505172 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.505148    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:59.505216 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.505203    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:59.533369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.532804    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:29:59.562676 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.562642    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:29:59.890744 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.890698    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"]
Oct 31 13:29:59.892855 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:29:59.892821    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa19833b_88b6_453e_81c8_4ce97c30669e.slice/crio-2f1184e530498da5eb479f2f861bc0834c50812fd5b5d22ea90d7e0e15c01688 WatchSource:0}: Error finding container 2f1184e530498da5eb479f2f861bc0834c50812fd5b5d22ea90d7e0e15c01688: Status 404 returned error can't find the container with id 2f1184e530498da5eb479f2f861bc0834c50812fd5b5d22ea90d7e0e15c01688
Oct 31 13:29:59.895867 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.895750    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:59.895961 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.895927    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:59.896007 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.895980    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:59.910342 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.910194    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"]
Oct 31 13:29:59.926446 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:29:59.926416    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1daa31f8_4e70_413f_9529_0a67eef57732.slice/crio-a7913f5e399d41ee3b46b8ae7e7f8e4d010f7d1240db921d537adefe90fb5b07 WatchSource:0}: Error finding container a7913f5e399d41ee3b46b8ae7e7f8e4d010f7d1240db921d537adefe90fb5b07: Status 404 returned error can't find the container with id a7913f5e399d41ee3b46b8ae7e7f8e4d010f7d1240db921d537adefe90fb5b07
Oct 31 13:29:59.933110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.933073    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:59.933223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.933159    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:59.933223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.933199    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:29:59.951041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.947944    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" event={"ID":"1daa31f8-4e70-413f-9529-0a67eef57732","Type":"ContainerStarted","Data":"a7913f5e399d41ee3b46b8ae7e7f8e4d010f7d1240db921d537adefe90fb5b07"}
Oct 31 13:29:59.951041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.950488    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" event={"ID":"6acc3b99-906c-429c-99f8-3aee4333eb66","Type":"ContainerStarted","Data":"3d418bde893a902938ba0daf48d9ec62bd6217fb88131bd8d9ee3bbcd16ac1c9"}
Oct 31 13:29:59.951041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.950534    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" event={"ID":"6acc3b99-906c-429c-99f8-3aee4333eb66","Type":"ContainerStarted","Data":"073caf68985e2a7da2f9c3c6c7e1e78d0a4ad852a3605a8a753f0c13ea54ed7b"}
Oct 31 13:29:59.953077 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.953051    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" event={"ID":"aa19833b-88b6-453e-81c8-4ce97c30669e","Type":"ContainerStarted","Data":"2f1184e530498da5eb479f2f861bc0834c50812fd5b5d22ea90d7e0e15c01688"}
Oct 31 13:29:59.981876 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:29:59.981824    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podStartSLOduration=1.981806008 podStartE2EDuration="1.981806008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:59.974096237 +0000 UTC m=+1599.592755402" watchObservedRunningTime="2026-10-31 13:29:59.981806008 +0000 UTC m=+1599.600465173"
Oct 31 13:30:00.152693 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:00.152601    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:30:00.155779 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:00.155750    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:30:00.156127 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:00.156073    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 13:30:01.038791 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:01.038747    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" event={"ID":"aa19833b-88b6-453e-81c8-4ce97c30669e","Type":"ContainerStarted","Data":"5857f4768517a5769b30be330ccf13c59e589f0860b7e3bcfdd977e9da83330d"}
Oct 31 13:30:01.128331 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:01.079667    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" event={"ID":"1daa31f8-4e70-413f-9529-0a67eef57732","Type":"ContainerStarted","Data":"79bed033ec609fdfd49624c999abf5b779e2be762f5a22595e5fb2ab17ece647"}
Oct 31 13:30:01.151685 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:01.151604    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podStartSLOduration=2.151579365 podStartE2EDuration="2.151579365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:01.072285159 +0000 UTC m=+1600.690944323" watchObservedRunningTime="2026-10-31 13:30:01.151579365 +0000 UTC m=+1600.770238529"
Oct 31 13:30:01.158188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:01.158118    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podStartSLOduration=2.158099027 podStartE2EDuration="2.158099027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:01.124995981 +0000 UTC m=+1600.743655146" watchObservedRunningTime="2026-10-31 13:30:01.158099027 +0000 UTC m=+1600.776758191"
Oct 31 13:30:01.186847 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:01.186812    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:30:01.187142 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:01.187110    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 13:30:01.540315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:01.539125    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:30:01.540315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:01.539344    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:30:01.540315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:01.539390    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 13:30:01.565258 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:01.565229    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:30:01.565634 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:01.565611    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:30:01.565785 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:01.565761    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 13:30:02.142320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:02.141564    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:02.158320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:02.154647    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:30:02.158320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:02.154707    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 13:30:02.444377 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:02.443398    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:02.543317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:02.542813    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:30:02.543317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:02.542872    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 13:30:02.578316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:02.566613    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:30:02.578316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:02.566668    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 13:30:02.589022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:02.582392    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:03.182318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:03.180965    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:30:03.182318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:03.181035    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 13:30:03.456372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:03.451390    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:03.456372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:03.452006    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:30:03.523317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:03.511447    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:03.549343 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:03.544448    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:30:03.559459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:03.553619    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 13:30:03.622619 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:03.593474    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:30:03.622619 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:03.593542    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 13:30:04.163730 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:04.163694    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:30:04.164014 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:04.163988    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 13:30:04.219314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:04.218889    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:30:04.778385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:04.517148    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:30:04.778385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:04.579452    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:30:04.778385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:04.579504    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 13:30:04.778385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:04.580195    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:30:04.778385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:04.580233    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 13:30:04.778385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:04.696838    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:04.852774 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:04.852727    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:04.853705 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:04.853672    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:04.896715 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:04.894486    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:04.896715 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:04.894787    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:30:05.029976 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.029837    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:05.099329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.097891    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:05.169009 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.168544    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:30:05.169009 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.168609    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 13:30:05.227928 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.227454    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:05.571327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.557618    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:30:05.571327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.557674    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 13:30:05.605319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.584846    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:30:05.605319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.584904    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 13:30:05.605319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.586908    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:05.605319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.588066    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:30:05.800318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.796515    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:30:05.800318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.799852    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:30:05.857495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.857396    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:05.858116 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.858082    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:30:05.942365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.931492    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:05.942365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.931846    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:30:05.942365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.872387    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:30:05.942365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.936159    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:30:05.942365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.936202    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:30:05.942365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:05.936238    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:30:06.011806 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.011763    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:30:06.050418 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.050371    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:06.076101 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.076069    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:30:06.101436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.098565    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:06.102244 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.102224    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:30:06.194356 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.191472    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:30:06.194356 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.191534    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 13:30:06.312376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.244639    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:06.644378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.620549    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:30:06.644378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.552240    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:30:06.644378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.623423    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 13:30:06.784214 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.784183    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:30:06.785224 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.627278    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:30:06.785459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.785429    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 13:30:06.802329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.796813    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:30:06.882319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.865375    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:06.947072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.929677    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:06.947360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.947323    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:30:06.947854 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:06.947828    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:30:07.079318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:07.050390    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:30:07.079318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:07.051493    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:30:07.079642 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:07.079503    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:07.102713 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:07.101758    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:30:07.211381 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:07.207721    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:30:07.211381 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:07.207778    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 13:30:07.785470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:07.785409    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:30:07.861676 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:07.822615    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:30:07.861676 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:07.822684    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 13:30:07.861676 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:07.835588    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:30:07.915321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:07.891901    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:30:07.960600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:07.960538    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:30:08.051527 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:07.995331    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:08.125492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.031574    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:30:08.140424 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.107159    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 13:30:08.187340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.123352    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:30:08.187340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.123414    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:30:08.187340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.154504    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:30:08.227642 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.217441    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:08.239228 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.220446    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:08.239228 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.221492    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:30:08.239228 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.235094    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 13:30:08.275512 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.275465    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:08.275800 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.275765    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs" podUID="13e81951-495d-4169-bdbb-f6484e5d4756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:08.556005 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.555966    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:30:08.566860 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.556354    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 13:30:08.585129 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.585092    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:30:08.585424 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.585394    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 13:30:08.797436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.797390    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:08.842621 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:08.842523    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:09.003896 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.003848    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:09.037465 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.037400    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:09.125275 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.125237    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:09.126121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.126086    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:09.155706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.155666    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:30:09.155976 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.155945    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 13:30:09.156604 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.156469    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:09.157001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.156503    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:30:09.157381 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.156525    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:09.223329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.222836    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Oct 31 13:30:09.223329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.222927    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Oct 31 13:30:09.240322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.238486    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:30:09.240322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.238871    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:09.540270 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.539781    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:30:09.540270 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.539839    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 13:30:09.620320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.596645    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:30:09.620320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.596705    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 13:30:09.827675 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.816489    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:09.974322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:09.892525    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:10.043312 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:10.042412    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:30:10.136459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:10.136410    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:10.136933 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:10.136904    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:10.154319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:10.137995    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:10.511384 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:10.180389    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:10.511384 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:10.180717    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:30:10.511384 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:10.180750    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:30:10.511384 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:10.180812    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:30:10.512517 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:10.249656    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:10.512978 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:10.249695    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:30:10.566320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:10.532638    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 13:30:10.801393 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:10.743420    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:30:10.801393 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:10.743485    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 13:30:10.801393 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:10.745777    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:30:10.801393 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:10.745821    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 13:30:11.002448 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.002402    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:30:11.035313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.029717    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:11.063368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.045461    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:30:11.063368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.047646    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:11.090810 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.090776    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:11.091088 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.091054    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:11.141366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.141317    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:11.167429 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.167394    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:30:11.167686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.167660    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 13:30:11.221344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.205263    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:11.425690 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.425644    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded"
Oct 31 13:30:11.517433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.517387    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:11.522339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.517666    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:30:11.562326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.517698    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:30:11.562326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.517717    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:30:11.562326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.517735    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:30:11.562326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.517782    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:11.634374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.634336    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:30:11.634654 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.634625    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 13:30:11.634924 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.634903    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:30:11.635076 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.635057    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 13:30:11.794623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:11.794524    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.047371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.043440    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.047371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.044813    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.050521 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.048847    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:30:12.153332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.145580    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.170511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.158913    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:30:12.170822 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.170779    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 13:30:12.576061 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.561224    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:30:12.576061 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.561281    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 13:30:12.576061 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.563209    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.576061 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.566277    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:30:12.576061 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.566590    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:30:12.576061 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.567646    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:30:12.576061 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.567680    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 13:30:12.576061 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.567728    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.672336 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.661852    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.672336 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.663758    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:30:12.722429 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:12.722380    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded"
Oct 31 13:30:13.592527 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.592480    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:13.694364 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.592649    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:13.736348 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.593405    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:13.813405 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.597037    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:13.816788 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.645338    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:30:13.835973 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.737159    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:30:13.836239 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.737217    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:30:13.836391 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.737272    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:30:13.836515 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.738387    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:30:13.836626 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.776404    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:30:13.836725 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.776464    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:30:13.836843 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.834644    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:13.836955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.772284    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:13.837051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.772363    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:13.869188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.869096    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 13:30:13.869779 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.869195    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 13:30:13.869779 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:13.869258    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 13:30:13.953915 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:30:13.946882    4660 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.385s"
Oct 31 13:30:14.030835 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.030805    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:30:14.032916 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.032889    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:30:14.036261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.036187    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:30:14.036561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.036540    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:30:14.061245 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.061219    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:30:14.063191 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.063170    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:30:14.075456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.075435    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:30:14.087464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.087065    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:30:14.087464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.087122    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:30:14.120655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.120631    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:30:14.153758 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.153718    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:30:14.168508 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.168453    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 13:30:14.168735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.154371    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:30:14.534365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.533720    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:30:14.534365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.533790    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 13:30:14.564426 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.564396    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:30:14.566251 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.565280    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:30:14.566479 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.566456    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 13:30:14.566912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:14.566895    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:30:15.152044 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:15.152009    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:30:15.152712 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:15.152664    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 13:30:16.024140 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:16.023890    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:30:16.031201 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:16.031173    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:30:16.163331 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:16.163260    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:30:16.165136 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:16.165107    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:30:16.575439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:16.575402    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:16.575598 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:16.575458    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:17.394673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:17.394615    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:30:17.397221 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:17.397192    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 13:30:20.733174 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:30:20.733132    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:30:21.213964 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:21.213930    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11116-282996"]
Oct 31 13:30:21.219782 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:21.219758    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-282996"
Oct 31 13:30:21.227584 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:21.227559    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-whw9x\""
Oct 31 13:30:21.231342 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:21.231287    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11116-282996"]
Oct 31 13:30:21.259867 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:21.259836    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs5cc\" (UniqueName: \"kubernetes.io/projected/9963d12f-2077-44ad-8c89-4eb33db2f38d-kube-api-access-bs5cc\") pod \"test-curl-11116-282996\" (UID: \"9963d12f-2077-44ad-8c89-4eb33db2f38d\") " pod="opendatahub/test-curl-11116-282996"
Oct 31 13:30:21.361101 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:21.361067    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs5cc\" (UniqueName: \"kubernetes.io/projected/9963d12f-2077-44ad-8c89-4eb33db2f38d-kube-api-access-bs5cc\") pod \"test-curl-11116-282996\" (UID: \"9963d12f-2077-44ad-8c89-4eb33db2f38d\") " pod="opendatahub/test-curl-11116-282996"
Oct 31 13:30:21.373928 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:21.373897    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs5cc\" (UniqueName: \"kubernetes.io/projected/9963d12f-2077-44ad-8c89-4eb33db2f38d-kube-api-access-bs5cc\") pod \"test-curl-11116-282996\" (UID: \"9963d12f-2077-44ad-8c89-4eb33db2f38d\") " pod="opendatahub/test-curl-11116-282996"
Oct 31 13:30:21.535008 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:21.534918    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-282996"
Oct 31 13:30:21.829829 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:30:21.829797    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9963d12f_2077_44ad_8c89_4eb33db2f38d.slice/crio-eac2561e2a0144e4b3c5f83bd2001dc28c6806a3ef6045a1893d70ff2980bc5f WatchSource:0}: Error finding container eac2561e2a0144e4b3c5f83bd2001dc28c6806a3ef6045a1893d70ff2980bc5f: Status 404 returned error can't find the container with id eac2561e2a0144e4b3c5f83bd2001dc28c6806a3ef6045a1893d70ff2980bc5f
Oct 31 13:30:21.831792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:21.831763    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11116-282996"]
Oct 31 13:30:21.832003 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:21.831975    4660 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:30:22.076432 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:22.076338    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11116-282996" event={"ID":"9963d12f-2077-44ad-8c89-4eb33db2f38d","Type":"ContainerStarted","Data":"eac2561e2a0144e4b3c5f83bd2001dc28c6806a3ef6045a1893d70ff2980bc5f"}
Oct 31 13:30:25.109410 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:25.109367    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11116-282996" event={"ID":"9963d12f-2077-44ad-8c89-4eb33db2f38d","Type":"ContainerStarted","Data":"812e4f9668e829694e095487caefadf18a7031487f4be91af350cb1de69e1057"}
Oct 31 13:30:25.129084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:25.129034    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11116-282996" podStartSLOduration=1.609636744 podStartE2EDuration="4.12899964s" totalImagesPullingTime="2.519362896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:25.126478546 +0000 UTC m=+1624.745137712" watchObservedRunningTime="2026-10-31 13:30:25.12899964 +0000 UTC m=+1624.747658803"
Oct 31 13:30:25.822278 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:25.822232    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11116-282996"]
Oct 31 13:30:25.828527 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:25.828490    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11116-282996"]
Oct 31 13:30:26.002026 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:26.001986    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11116-4fe1cf"]
Oct 31 13:30:26.002735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:26.002713    4660 cpu_manager.go:414] "containerMap: removing container" podUID="9963d12f-2077-44ad-8c89-4eb33db2f38d" containerName="test-curl-11116-282996"
Oct 31 13:30:26.002735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:26.002735    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9963d12f-2077-44ad-8c89-4eb33db2f38d" containerName="test-curl-11116-282996"
Oct 31 13:30:26.002940 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:26.002926    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="9963d12f-2077-44ad-8c89-4eb33db2f38d" containerName="test-curl-11116-282996"
Oct 31 13:30:26.006824 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:26.006804    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-4fe1cf"
Oct 31 13:30:26.014442 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:26.014417    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11116-4fe1cf"]
Oct 31 13:30:26.145339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:26.143354    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqwl8\" (UniqueName: \"kubernetes.io/projected/a8b04eae-9311-49a4-921d-cbdc60ed32e5-kube-api-access-pqwl8\") pod \"test-curl-11116-4fe1cf\" (UID: \"a8b04eae-9311-49a4-921d-cbdc60ed32e5\") " pod="opendatahub/test-curl-11116-4fe1cf"
Oct 31 13:30:26.244914 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:26.244866    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqwl8\" (UniqueName: \"kubernetes.io/projected/a8b04eae-9311-49a4-921d-cbdc60ed32e5-kube-api-access-pqwl8\") pod \"test-curl-11116-4fe1cf\" (UID: \"a8b04eae-9311-49a4-921d-cbdc60ed32e5\") " pod="opendatahub/test-curl-11116-4fe1cf"
Oct 31 13:30:26.258443 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:26.258411    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqwl8\" (UniqueName: \"kubernetes.io/projected/a8b04eae-9311-49a4-921d-cbdc60ed32e5-kube-api-access-pqwl8\") pod \"test-curl-11116-4fe1cf\" (UID: \"a8b04eae-9311-49a4-921d-cbdc60ed32e5\") " pod="opendatahub/test-curl-11116-4fe1cf"
Oct 31 13:30:26.326727 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:26.326695    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-4fe1cf"
Oct 31 13:30:26.619276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:26.619243    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11116-4fe1cf"]
Oct 31 13:30:26.620760 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:30:26.620731    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8b04eae_9311_49a4_921d_cbdc60ed32e5.slice/crio-fb410155f21684756f783aef2f3bb3277741b7acdda84347415c76077421ede0 WatchSource:0}: Error finding container fb410155f21684756f783aef2f3bb3277741b7acdda84347415c76077421ede0: Status 404 returned error can't find the container with id fb410155f21684756f783aef2f3bb3277741b7acdda84347415c76077421ede0
Oct 31 13:30:27.131265 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:27.131222    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11116-4fe1cf" event={"ID":"a8b04eae-9311-49a4-921d-cbdc60ed32e5","Type":"ContainerStarted","Data":"260d4aca79ecaab5c55f2d9c6fda55c7ba9df54b615388d2628314ad7f2b7f92"}
Oct 31 13:30:27.131477 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:27.131274    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11116-4fe1cf" event={"ID":"a8b04eae-9311-49a4-921d-cbdc60ed32e5","Type":"ContainerStarted","Data":"fb410155f21684756f783aef2f3bb3277741b7acdda84347415c76077421ede0"}
Oct 31 13:30:27.131477 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:27.131384    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11116-282996" podUID="9963d12f-2077-44ad-8c89-4eb33db2f38d" containerName="test-curl-11116-282996" containerID="cri-o://812e4f9668e829694e095487caefadf18a7031487f4be91af350cb1de69e1057" gracePeriod=2
Oct 31 13:30:27.149197 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:27.149126    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11116-4fe1cf" podStartSLOduration=2.149109492 podStartE2EDuration="2.149109492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:27.148397711 +0000 UTC m=+1626.767056890" watchObservedRunningTime="2026-10-31 13:30:27.149109492 +0000 UTC m=+1626.767768658"
Oct 31 13:30:27.900039 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:27.900004    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11116-4fe1cf"]
Oct 31 13:30:27.908279 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:27.908251    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11116-4fe1cf"]
Oct 31 13:30:29.178002 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:29.177969    4660 generic.go:356] "Generic (PLEG): container finished" podID="9963d12f-2077-44ad-8c89-4eb33db2f38d" containerID="812e4f9668e829694e095487caefadf18a7031487f4be91af350cb1de69e1057" exitCode=137
Oct 31 13:30:29.178331 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:29.178180    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11116-4fe1cf" podUID="a8b04eae-9311-49a4-921d-cbdc60ed32e5" containerName="test-curl-11116-4fe1cf" containerID="cri-o://260d4aca79ecaab5c55f2d9c6fda55c7ba9df54b615388d2628314ad7f2b7f92" gracePeriod=2
Oct 31 13:30:29.672976 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:29.672945    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-282996"
Oct 31 13:30:29.739051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:29.739014    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9963d12f-2077-44ad-8c89-4eb33db2f38d-kube-api-access-bs5cc\" (UniqueName: \"kubernetes.io/projected/9963d12f-2077-44ad-8c89-4eb33db2f38d-kube-api-access-bs5cc\") pod \"9963d12f-2077-44ad-8c89-4eb33db2f38d\" (UID: \"9963d12f-2077-44ad-8c89-4eb33db2f38d\") "
Oct 31 13:30:29.742821 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:29.742788    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9963d12f-2077-44ad-8c89-4eb33db2f38d-kube-api-access-bs5cc" pod "9963d12f-2077-44ad-8c89-4eb33db2f38d" (UID: "9963d12f-2077-44ad-8c89-4eb33db2f38d"). InnerVolumeSpecName "kube-api-access-bs5cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:30:29.840988 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:29.840947    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs5cc\" (UniqueName: \"kubernetes.io/projected/9963d12f-2077-44ad-8c89-4eb33db2f38d-kube-api-access-bs5cc\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:30.189901 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:30.189873    4660 scope.go:122] "RemoveContainer" containerID="812e4f9668e829694e095487caefadf18a7031487f4be91af350cb1de69e1057"
Oct 31 13:30:30.190408 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:30.190031    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-282996"
Oct 31 13:30:30.219817 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:30:30.219774    4660 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11116-282996\" is forbidden: User \"system:node:ip-10-0-137-77.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-77.us-east-2.compute.internal' and this object" podUID="9963d12f-2077-44ad-8c89-4eb33db2f38d" pod="opendatahub/test-curl-11116-282996"
Oct 31 13:30:30.530015 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:30:30.529922    4660 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11116-282996\" is forbidden: User \"system:node:ip-10-0-137-77.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-77.us-east-2.compute.internal' and this object" podUID="9963d12f-2077-44ad-8c89-4eb33db2f38d" pod="opendatahub/test-curl-11116-282996"
Oct 31 13:30:30.530223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:30.530197    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9963d12f-2077-44ad-8c89-4eb33db2f38d" path="/var/lib/kubelet/pods/9963d12f-2077-44ad-8c89-4eb33db2f38d/volumes"
Oct 31 13:30:31.758195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:31.758172    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-4fe1cf"
Oct 31 13:30:31.802714 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:31.802680    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b04eae-9311-49a4-921d-cbdc60ed32e5-kube-api-access-pqwl8\" (UniqueName: \"kubernetes.io/projected/a8b04eae-9311-49a4-921d-cbdc60ed32e5-kube-api-access-pqwl8\") pod \"a8b04eae-9311-49a4-921d-cbdc60ed32e5\" (UID: \"a8b04eae-9311-49a4-921d-cbdc60ed32e5\") "
Oct 31 13:30:31.804916 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:31.804891    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b04eae-9311-49a4-921d-cbdc60ed32e5-kube-api-access-pqwl8" pod "a8b04eae-9311-49a4-921d-cbdc60ed32e5" (UID: "a8b04eae-9311-49a4-921d-cbdc60ed32e5"). InnerVolumeSpecName "kube-api-access-pqwl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:30:31.906175 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:31.906134    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqwl8\" (UniqueName: \"kubernetes.io/projected/a8b04eae-9311-49a4-921d-cbdc60ed32e5-kube-api-access-pqwl8\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:32.310887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:32.310789    4660 generic.go:356] "Generic (PLEG): container finished" podID="a8b04eae-9311-49a4-921d-cbdc60ed32e5" containerID="260d4aca79ecaab5c55f2d9c6fda55c7ba9df54b615388d2628314ad7f2b7f92" exitCode=137
Oct 31 13:30:32.310887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:32.310842    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-4fe1cf"
Oct 31 13:30:32.311113 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:32.310899    4660 scope.go:122] "RemoveContainer" containerID="260d4aca79ecaab5c55f2d9c6fda55c7ba9df54b615388d2628314ad7f2b7f92"
Oct 31 13:30:32.322828 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:30:32.322796    4660 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11116-4fe1cf\" is forbidden: User \"system:node:ip-10-0-137-77.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-77.us-east-2.compute.internal' and this object" podUID="a8b04eae-9311-49a4-921d-cbdc60ed32e5" pod="opendatahub/test-curl-11116-4fe1cf"
Oct 31 13:30:32.323915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:32.323892    4660 scope.go:122] "RemoveContainer" containerID="260d4aca79ecaab5c55f2d9c6fda55c7ba9df54b615388d2628314ad7f2b7f92"
Oct 31 13:30:32.324182 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:30:32.324159    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"260d4aca79ecaab5c55f2d9c6fda55c7ba9df54b615388d2628314ad7f2b7f92\": container with ID starting with 260d4aca79ecaab5c55f2d9c6fda55c7ba9df54b615388d2628314ad7f2b7f92 not found: ID does not exist" containerID="260d4aca79ecaab5c55f2d9c6fda55c7ba9df54b615388d2628314ad7f2b7f92"
Oct 31 13:30:32.324253 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:32.324193    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260d4aca79ecaab5c55f2d9c6fda55c7ba9df54b615388d2628314ad7f2b7f92"} err="failed to get container status \"260d4aca79ecaab5c55f2d9c6fda55c7ba9df54b615388d2628314ad7f2b7f92\": rpc error: code = NotFound desc = could not find container \"260d4aca79ecaab5c55f2d9c6fda55c7ba9df54b615388d2628314ad7f2b7f92\": container with ID starting with 260d4aca79ecaab5c55f2d9c6fda55c7ba9df54b615388d2628314ad7f2b7f92 not found: ID does not exist"
Oct 31 13:30:32.528193 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:32.528157    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b04eae-9311-49a4-921d-cbdc60ed32e5" path="/var/lib/kubelet/pods/a8b04eae-9311-49a4-921d-cbdc60ed32e5/volumes"
Oct 31 13:30:34.441984 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.441952    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"]
Oct 31 13:30:34.442642 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.442626    4660 cpu_manager.go:414] "containerMap: removing container" podUID="a8b04eae-9311-49a4-921d-cbdc60ed32e5" containerName="test-curl-11116-4fe1cf"
Oct 31 13:30:34.442699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.442643    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b04eae-9311-49a4-921d-cbdc60ed32e5" containerName="test-curl-11116-4fe1cf"
Oct 31 13:30:34.442775 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.442766    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b04eae-9311-49a4-921d-cbdc60ed32e5" containerName="test-curl-11116-4fe1cf"
Oct 31 13:30:34.447681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.447662    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.454117 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.454091    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8439b78a-openshift-default-dockercfg-tkc5p\""
Oct 31 13:30:34.469342 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.469091    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"]
Oct 31 13:30:34.575315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.569073    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-workload-certs\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.575315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.569214    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231f47ca-89fc-4370-853f-b0d788524589-istio-token\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.575315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.569309    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfmmz\" (UniqueName: \"kubernetes.io/projected/231f47ca-89fc-4370-853f-b0d788524589-kube-api-access-hfmmz\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.575315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.569364    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-credential-socket\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.575315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.569438    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-workload-socket\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.575315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.569483    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-istio-envoy\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.575315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.569586    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-istio-data\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.575315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.569632    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231f47ca-89fc-4370-853f-b0d788524589-istiod-ca-cert\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.575315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.569934    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231f47ca-89fc-4370-853f-b0d788524589-istio-podinfo\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.671570 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.671518    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-workload-socket\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.671769 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.671578    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-istio-envoy\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.671769 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.671663    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-istio-data\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.671769 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.671696    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231f47ca-89fc-4370-853f-b0d788524589-istiod-ca-cert\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.671942 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.671853    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231f47ca-89fc-4370-853f-b0d788524589-istio-podinfo\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.671942 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.671920    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-workload-certs\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.672049 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.672003    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231f47ca-89fc-4370-853f-b0d788524589-istio-token\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.672102 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.672062    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfmmz\" (UniqueName: \"kubernetes.io/projected/231f47ca-89fc-4370-853f-b0d788524589-kube-api-access-hfmmz\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.672152 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.672113    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-credential-socket\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.672446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.672421    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-istio-data\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.672560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.672545    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-credential-socket\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.672767 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.672742    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-workload-certs\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.672844 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.672749    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-workload-socket\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.673410 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.673389    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231f47ca-89fc-4370-853f-b0d788524589-istiod-ca-cert\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.675715 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.675694    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-istio-envoy\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.676055 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.676032    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231f47ca-89fc-4370-853f-b0d788524589-istio-podinfo\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.692907 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.692835    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231f47ca-89fc-4370-853f-b0d788524589-istio-token\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.693247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.693218    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfmmz\" (UniqueName: \"kubernetes.io/projected/231f47ca-89fc-4370-853f-b0d788524589-kube-api-access-hfmmz\") pod \"e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") " pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:34.773840 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:34.773195    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:35.260484 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:35.251608    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"]
Oct 31 13:30:35.266329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:35.262976    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:30:35.266329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:35.263063    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:30:35.266329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:35.263107    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:30:35.361034 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:35.360995    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d" event={"ID":"231f47ca-89fc-4370-853f-b0d788524589","Type":"ContainerStarted","Data":"4495691bbb86572054435bfe5b895e64ae5109a738eafe36e384104b6414a100"}
Oct 31 13:30:36.374135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:36.374090    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d" event={"ID":"231f47ca-89fc-4370-853f-b0d788524589","Type":"ContainerStarted","Data":"c80a37c3ea4fd0651136a75d1f1229c4b6dbceaf62cc2f16af8c1c03725046e6"}
Oct 31 13:30:36.431737 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:36.431672    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d" podStartSLOduration=2.4316496499999998 podStartE2EDuration="2.43164965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:36.405081177 +0000 UTC m=+1636.023740342" watchObservedRunningTime="2026-10-31 13:30:36.43164965 +0000 UTC m=+1636.050308816"
Oct 31 13:30:36.775313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:36.775215    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:36.781396 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:36.781371    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:37.400977 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:37.400027    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:37.401595 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:37.401104    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:30:39.826015 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:39.817683    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:30:39.952330 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:39.949986    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:30:39.954050 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:39.953049    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:40.132370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:40.125390    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:40.136488 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:40.136443    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:40.161574 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:40.161535    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:40.854317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:40.825425    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:40.934353 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:40.930861    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:41.140380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:40.959384    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:41.140380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:41.019908    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:30:41.140380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:41.114531    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:30:41.140380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:41.114591    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:30:41.142218 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:41.140818    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:41.142218 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:41.141156    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:41.142218 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:41.141707    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:30:41.175393 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:41.143218    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:41.482591 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:41.463067    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:41.669317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:41.644676    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:41.814624 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:41.552019    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:30:41.815611 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:41.552458    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded"
Oct 31 13:30:41.815611 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:41.531836    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:30:41.815611 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:41.700497    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:30:41.815611 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:41.773110    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:41.815611 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:41.813998    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:30:41.858490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:41.858433    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:42.030317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:42.023067    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:30:42.180319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:42.164158    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:30:42.180319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:42.165424    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:42.180319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:42.165743    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:30:42.180319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:42.167610    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:30:42.180319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:42.167674    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:30:42.249644 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:42.249596    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:42.291635 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:42.257437    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:30:42.291635 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:42.278992    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:42.322260 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:42.316478    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:30:42.833320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:42.813257    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:43.031591 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.031542    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:43.039772 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.039734    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:43.059334 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.040726    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:43.117404 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.113571    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:43.178982 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.178924    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:30:43.180765 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.180733    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:43.200035 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.199987    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:30:43.233313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.205990    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:30:43.234305 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.234262    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:30:43.234530 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.234513    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:30:43.298319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.294948    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:43.315364 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.303483    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:30:43.376154 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.375343    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:43.395888 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.395856    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"]
Oct 31 13:30:43.452600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.452570    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"
Oct 31 13:30:43.598850 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.598794    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:43.599966 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.599942    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8439b78a-dockercfg-q6dt7\""
Oct 31 13:30:43.614184 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.613611    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8439b78a\""
Oct 31 13:30:43.726414 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.719732    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f5af14-f5de-4072-aeee-d32d4aaf1bde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq\" (UID: \"95f5af14-f5de-4072-aeee-d32d4aaf1bde\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"
Oct 31 13:30:43.726414 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.719931    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl448\" (UniqueName: \"kubernetes.io/projected/95f5af14-f5de-4072-aeee-d32d4aaf1bde-kube-api-access-pl448\") pod \"payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq\" (UID: \"95f5af14-f5de-4072-aeee-d32d4aaf1bde\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"
Oct 31 13:30:43.850793 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.850741    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl448\" (UniqueName: \"kubernetes.io/projected/95f5af14-f5de-4072-aeee-d32d4aaf1bde-kube-api-access-pl448\") pod \"payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq\" (UID: \"95f5af14-f5de-4072-aeee-d32d4aaf1bde\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"
Oct 31 13:30:43.851394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.851353    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f5af14-f5de-4072-aeee-d32d4aaf1bde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq\" (UID: \"95f5af14-f5de-4072-aeee-d32d4aaf1bde\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"
Oct 31 13:30:43.884336 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:43.884277    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f5af14-f5de-4072-aeee-d32d4aaf1bde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq\" (UID: \"95f5af14-f5de-4072-aeee-d32d4aaf1bde\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"
Oct 31 13:30:44.057596 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:44.057490    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:30:44.110469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:44.110429    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:30:44.111026 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:44.110571    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:44.138328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:44.118468    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:44.139189 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:44.139148    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"]
Oct 31 13:30:44.195933 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:44.195887    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:30:44.196574 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:44.196541    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:44.253651 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:44.236558    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:44.302528 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:44.302486    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:44.400285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:44.377187    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:44.400285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:44.377597    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:30:44.425319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:44.421699    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl448\" (UniqueName: \"kubernetes.io/projected/95f5af14-f5de-4072-aeee-d32d4aaf1bde-kube-api-access-pl448\") pod \"payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq\" (UID: \"95f5af14-f5de-4072-aeee-d32d4aaf1bde\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"
Oct 31 13:30:44.563635 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:44.563595    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"
Oct 31 13:30:45.190087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:45.190034    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:45.235026 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:45.234915    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:30:45.263619 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:45.263576    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:45.334371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:45.316840    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:30:45.337280 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:45.337235    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:45.365441 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:45.365375    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:30:45.400720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:45.400669    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:30:45.500001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:45.499898    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:30:45.640456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:45.640401    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:45.641085 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:45.641052    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:30:45.907719 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:45.907676    4660 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:45.908028 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:45.907994    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:46.195488 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:46.195385    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:46.237425 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:46.237375    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:46.237919 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:46.237894    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:30:46.269663 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:46.269616    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:46.307087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:46.307039    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:46.402986 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:46.386896    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:46.402986 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:46.387251    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:30:46.451151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:46.415899    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:46.579702 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:46.579655    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:46.643971 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:46.643918    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:47.256650 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.256597    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:30:47.282588 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.260974    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:47.282588 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.279474    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:30:47.282588 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.280417    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:47.307274 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.307235    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:47.307637 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.307594    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:47.307904 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.307810    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:30:47.309175 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.309156    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:30:47.327962 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.326754    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:47.327962 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.326818    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:47.344237 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.344073    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:47.344237 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.344138    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:47.484373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.375472    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"]
Oct 31 13:30:47.484373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.408161    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:47.484373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.409249    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:30:47.484373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.448430    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:47.593799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.581919    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:47.719350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.694431    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:47.719891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.719664    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:47.719891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:47.719755    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:48.084607 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:48.084578    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:30:48.170812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:48.168205    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:30:48.208573 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:48.208548    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:30:48.296357 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:48.280088    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:48.356729 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:48.356643    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:48.377381 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:48.366163    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:48.409319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:48.408456    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:48.464318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:48.451406    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:48.464318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:48.451944    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:48.464318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:48.451991    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:48.504859 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:48.504832    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:30:48.684368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:48.518737    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:30:49.244977 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:49.244881    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Oct 31 13:30:49.307842 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:49.294572    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:30:49.419334 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:49.412550    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:49.511368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:49.510780    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:30:49.632843 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:49.612420    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:49.632843 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:49.624703    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:30:49.648256 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:49.648227    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:30:49.715310 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:49.705213    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq" event={"ID":"95f5af14-f5de-4072-aeee-d32d4aaf1bde","Type":"ContainerStarted","Data":"c3ef3acae9188b56d918d209ce17fc74e10fb4407011d2e12e29a337db034bb0"}
Oct 31 13:30:49.715310 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:49.705333    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"
Oct 31 13:30:49.759949 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:49.759915    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:30:49.776283 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:49.776147    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq" podStartSLOduration=7.776128027 podStartE2EDuration="7.776128027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:49.759063678 +0000 UTC m=+1649.377722837" watchObservedRunningTime="2026-10-31 13:30:49.776128027 +0000 UTC m=+1649.394787193"
Oct 31 13:30:49.795994 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:49.795638    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:30:50.746718 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:30:50.746680    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:30:50.757378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:50.757345    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq" event={"ID":"95f5af14-f5de-4072-aeee-d32d4aaf1bde","Type":"ContainerStarted","Data":"caeecba116583e313c933f6f186956c44b5c5482fc93ab0306a2280f418321e5"}
Oct 31 13:30:52.249744 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.247423    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"]
Oct 31 13:30:52.252130 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.252105    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.286776 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.286746    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a66ae960-openshift-default-dockercfg-vxzcs\""
Oct 31 13:30:52.292151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.292118    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"]
Oct 31 13:30:52.304812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.304777    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9926d7e7-5939-4e23-85b0-28682993eab9-istiod-ca-cert\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.304959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.304887    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9926d7e7-5939-4e23-85b0-28682993eab9-istio-podinfo\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.304959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.304927    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-credential-socket\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.305083 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.304987    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-istio-envoy\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.305155 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.305084    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9926d7e7-5939-4e23-85b0-28682993eab9-istio-token\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.305155 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.305142    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-istio-data\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.305270 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.305181    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-workload-certs\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.305270 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.305223    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-workload-socket\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.305429 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.305323    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mpdm\" (UniqueName: \"kubernetes.io/projected/9926d7e7-5939-4e23-85b0-28682993eab9-kube-api-access-7mpdm\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.406449 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.406413    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9926d7e7-5939-4e23-85b0-28682993eab9-istio-token\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.406632 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.406485    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-istio-data\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.406632 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.406525    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-workload-certs\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.406632 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.406569    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-workload-socket\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.406808 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.406646    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mpdm\" (UniqueName: \"kubernetes.io/projected/9926d7e7-5939-4e23-85b0-28682993eab9-kube-api-access-7mpdm\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.406808 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.406734    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9926d7e7-5939-4e23-85b0-28682993eab9-istiod-ca-cert\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.406915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.406829    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9926d7e7-5939-4e23-85b0-28682993eab9-istio-podinfo\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.406915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.406871    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-credential-socket\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.407036 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.406937    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-istio-envoy\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.407036 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.406963    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-istio-data\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.407568 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.407542    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9926d7e7-5939-4e23-85b0-28682993eab9-istiod-ca-cert\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.407689 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.407553    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-workload-socket\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.407752 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.407703    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-workload-certs\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.407845 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.407804    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-credential-socket\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.409810 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.409786    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-istio-envoy\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.410028 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.409821    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9926d7e7-5939-4e23-85b0-28682993eab9-istio-podinfo\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.435999 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.421828    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9926d7e7-5939-4e23-85b0-28682993eab9-istio-token\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.435999 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.421970    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mpdm\" (UniqueName: \"kubernetes.io/projected/9926d7e7-5939-4e23-85b0-28682993eab9-kube-api-access-7mpdm\") pod \"e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") " pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.599744 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.598608    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:52.976791 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.976766    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"]
Oct 31 13:30:52.982470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.982439    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:30:52.982574 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.982500    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:30:52.982574 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:52.982541    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:30:53.892927 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:53.892886    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh" event={"ID":"9926d7e7-5939-4e23-85b0-28682993eab9","Type":"ContainerStarted","Data":"532133cb4dadcaa80a18eedf5ac336b59ce2d6e2dec5463671c052979d9478f8"}
Oct 31 13:30:53.893607 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:53.893574    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh" event={"ID":"9926d7e7-5939-4e23-85b0-28682993eab9","Type":"ContainerStarted","Data":"e6d34cd4377641cc7893aa5b8662174701b3de1b79ae4bb5c8b164fd43c46361"}
Oct 31 13:30:53.942353 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:53.931902    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh" podStartSLOduration=1.93188262 podStartE2EDuration="1.93188262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:53.93074753 +0000 UTC m=+1653.549406705" watchObservedRunningTime="2026-10-31 13:30:53.93188262 +0000 UTC m=+1653.550541783"
Oct 31 13:30:54.600912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:54.600884    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:54.607821 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:54.607764    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:54.901095 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:54.901061    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:30:54.902233 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:30:54.902211    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:31:00.390392 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.390282    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"]
Oct 31 13:31:00.400852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.400828    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.411345 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.410620    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a8c1600c-openshift-default-dockercfg-rmx7z\""
Oct 31 13:31:00.414327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.414282    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"]
Oct 31 13:31:00.637950 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.633572    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqr8q\" (UniqueName: \"kubernetes.io/projected/49f9647f-e908-4ede-bb66-683c8f83f0e1-kube-api-access-kqr8q\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.641355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.641270    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-credential-socket\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.641604 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.641585    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-data\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.641766 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.641751    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-workload-certs\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.641932 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.641918    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-envoy\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.642109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.642096    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49f9647f-e908-4ede-bb66-683c8f83f0e1-istiod-ca-cert\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.642405 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.642387    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-token\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.642577 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.642562    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-podinfo\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.642701 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.642689    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-workload-socket\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.754275 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.754233    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqr8q\" (UniqueName: \"kubernetes.io/projected/49f9647f-e908-4ede-bb66-683c8f83f0e1-kube-api-access-kqr8q\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.755075 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.755010    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-credential-socket\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.755714 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.755690    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-data\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.756361 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.756343    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-workload-certs\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.756796 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.756781    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-envoy\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.757382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.757360    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49f9647f-e908-4ede-bb66-683c8f83f0e1-istiod-ca-cert\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.758411 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.758391    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-token\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.758919 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.758899    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-podinfo\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.759548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.758170    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49f9647f-e908-4ede-bb66-683c8f83f0e1-istiod-ca-cert\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.759684 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.756102    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-data\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.760073 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.756731    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-workload-certs\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.760194 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.755635    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-credential-socket\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.760324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.760021    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-workload-socket\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.760433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.759520    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-workload-socket\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.791479 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.789810    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqr8q\" (UniqueName: \"kubernetes.io/projected/49f9647f-e908-4ede-bb66-683c8f83f0e1-kube-api-access-kqr8q\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.794203 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.792631    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-token\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.801812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.801639    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-envoy\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:00.807514 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:00.804882    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-podinfo\") pod \"e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") " pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:01.062556 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:01.062456    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:01.427274 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:31:01.427244    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f9647f_e908_4ede_bb66_683c8f83f0e1.slice/crio-c2ab8030693ef48adbcf0b19a5348fd773a4aa0bc326b17a3cb47d885d42f4ce WatchSource:0}: Error finding container c2ab8030693ef48adbcf0b19a5348fd773a4aa0bc326b17a3cb47d885d42f4ce: Status 404 returned error can't find the container with id c2ab8030693ef48adbcf0b19a5348fd773a4aa0bc326b17a3cb47d885d42f4ce
Oct 31 13:31:01.429255 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:01.429179    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"]
Oct 31 13:31:01.430699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:01.430661    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:31:01.430786 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:01.430749    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:31:01.430829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:01.430790    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:31:01.807082 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:01.804702    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"
Oct 31 13:31:02.001370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:02.000845    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5" event={"ID":"49f9647f-e908-4ede-bb66-683c8f83f0e1","Type":"ContainerStarted","Data":"1dce5c0208b70a73c24d94ca1081a813b03e9aef4f6b29f5c62a9a7516f8db71"}
Oct 31 13:31:02.001370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:02.000911    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5" event={"ID":"49f9647f-e908-4ede-bb66-683c8f83f0e1","Type":"ContainerStarted","Data":"c2ab8030693ef48adbcf0b19a5348fd773a4aa0bc326b17a3cb47d885d42f4ce"}
Oct 31 13:31:02.038175 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:02.036434    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5" podStartSLOduration=2.036420707 podStartE2EDuration="2.036420707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:02.036092521 +0000 UTC m=+1661.654751681" watchObservedRunningTime="2026-10-31 13:31:02.036420707 +0000 UTC m=+1661.655079870"
Oct 31 13:31:02.063445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:02.063412    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:02.064954 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:02.064928    4660 patch_prober.go:29] interesting pod/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:31:02.065081 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:02.064979    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5" podUID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 13:31:03.069488 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:03.069453    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:04.071317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:04.055379    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:04.083320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:04.074511    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:31:05.562315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.561304    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"]
Oct 31 13:31:05.569459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.569436    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.570623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.570462    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"]
Oct 31 13:31:05.575089 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.575059    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 13:31:05.613054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.611256    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97903974-f732-4729-9d22-d4d319663078-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.613054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.611379    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97903974-f732-4729-9d22-d4d319663078-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.613054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.611409    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97903974-f732-4729-9d22-d4d319663078-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.613054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.611556    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97903974-f732-4729-9d22-d4d319663078-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.613054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.611611    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb8vq\" (UniqueName: \"kubernetes.io/projected/97903974-f732-4729-9d22-d4d319663078-kube-api-access-mb8vq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.613054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.611668    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97903974-f732-4729-9d22-d4d319663078-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.613054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.611773    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97903974-f732-4729-9d22-d4d319663078-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.675269 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.675235    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 13:31:05.675269 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.675235    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 13:31:05.675489 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.675326    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zfm9b\""
Oct 31 13:31:05.714054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.713319    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97903974-f732-4729-9d22-d4d319663078-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.714054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.713665    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97903974-f732-4729-9d22-d4d319663078-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.714054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.713703    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97903974-f732-4729-9d22-d4d319663078-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.714054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.713857    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97903974-f732-4729-9d22-d4d319663078-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.714054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.713919    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb8vq\" (UniqueName: \"kubernetes.io/projected/97903974-f732-4729-9d22-d4d319663078-kube-api-access-mb8vq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.714054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.713979    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97903974-f732-4729-9d22-d4d319663078-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.714516 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.714079    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97903974-f732-4729-9d22-d4d319663078-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.719500 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.719476    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97903974-f732-4729-9d22-d4d319663078-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.722126 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.722103    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97903974-f732-4729-9d22-d4d319663078-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.737822 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.737764    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97903974-f732-4729-9d22-d4d319663078-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.749179 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.749128    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97903974-f732-4729-9d22-d4d319663078-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.749995 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.749970    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97903974-f732-4729-9d22-d4d319663078-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.750531 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.750491    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97903974-f732-4729-9d22-d4d319663078-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.752922 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.752899    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb8vq\" (UniqueName: \"kubernetes.io/projected/97903974-f732-4729-9d22-d4d319663078-kube-api-access-mb8vq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-59fjz\" (UID: \"97903974-f732-4729-9d22-d4d319663078\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:05.889724 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:05.889695    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:06.294509 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:06.294406    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"]
Oct 31 13:31:06.295432 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:31:06.295403    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97903974_f732_4729_9d22_d4d319663078.slice/crio-8bcacdca63e8a1eaf61d482ae8e537af682848048b66629ee95d5dcd2f822edf WatchSource:0}: Error finding container 8bcacdca63e8a1eaf61d482ae8e537af682848048b66629ee95d5dcd2f822edf: Status 404 returned error can't find the container with id 8bcacdca63e8a1eaf61d482ae8e537af682848048b66629ee95d5dcd2f822edf
Oct 31 13:31:07.126589 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:07.126531    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz" event={"ID":"97903974-f732-4729-9d22-d4d319663078","Type":"ContainerStarted","Data":"8bcacdca63e8a1eaf61d482ae8e537af682848048b66629ee95d5dcd2f822edf"}
Oct 31 13:31:10.563767 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:10.563729    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:31:10.564119 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:10.563798    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:31:11.187171 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:11.187096    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz" event={"ID":"97903974-f732-4729-9d22-d4d319663078","Type":"ContainerStarted","Data":"4891aa6528e3ba9d1725d415b0cb51426791e63a52de6ebb0c7c599febdaf416"}
Oct 31 13:31:11.187393 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:11.187258    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:11.190691 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:11.190661    4660 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-59fjz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:31:11.190826 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:11.190741    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz" podUID="97903974-f732-4729-9d22-d4d319663078" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:31:12.197101 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:12.197076    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz"
Oct 31 13:31:12.228923 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:12.228867    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-59fjz" podStartSLOduration=2.9925525410000002 podStartE2EDuration="7.228848563s" totalImagesPullingTime="4.236296022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:11.226640985 +0000 UTC m=+1670.845300148" watchObservedRunningTime="2026-10-31 13:31:12.228848563 +0000 UTC m=+1671.847507730"
Oct 31 13:31:18.982318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:18.981966    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:19.111319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:19.111153    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:19.152918 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:19.136968    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:19.206317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:19.189043    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:19.231314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:19.220532    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:19.275377 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:19.274683    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:31:19.301318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:19.300399    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:19.328347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:19.316380    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:31:19.548919 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:19.545546    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:31:19.659328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:19.656688    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:20.110317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.110243    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:20.208348 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.199230    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded"
Oct 31 13:31:20.244316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.224436    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:31:20.244316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.227379    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:31:20.244316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.227895    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:31:20.244316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.230385    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:31:20.244316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.231050    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:31:20.268384 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.250983    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:20.276416 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.276378    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:20.277128 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.277106    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:31:20.383368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.339471    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:31:20.383368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.346363    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:20.383368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.363416    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:31:20.383368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.364459    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:31:20.551374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.551322    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:20.557357 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.552079    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:31:20.851375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:20.809955    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:21.074008 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.073967    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:21.183138 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.114478    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:21.260392 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.244363    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:31:21.261224 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.261196    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:31:21.308329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.283390    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:21.308329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.283645    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:31:21.308329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.283910    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:21.337410 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.322578    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:31:21.337410 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.322692    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:31:21.373459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.373412    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:21.383326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.375648    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:21.423322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.421330    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:21.423607 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.423580    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:31:21.461900 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.440611    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:31:21.485320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.474028    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:31:21.555968 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.555916    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:21.625494 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:31:21.625451    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:31:21.747169 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:21.747086    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:31:22.355175 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:22.355121    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:22.378083 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:22.378034    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:31:22.399318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:22.380137    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded"
Oct 31 13:31:22.434665 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:22.399863    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:31:22.436635 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:22.400691    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:31:22.511316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:22.434534    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:22.610954 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:22.610857    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:22.630850 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:22.630802    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:31:22.747516 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:22.747467    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:22.806326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:22.766998    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:31:22.827448 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:22.792921    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:31:22.897964 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:22.793519    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:31:23.152629 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:23.152573    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:31:23.153479 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:23.153459    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:31:23.363346 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:23.359859    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:23.450520 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:23.450406    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:23.518303 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:23.508912    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:23.536703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:23.520594    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:23.618225 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:23.618198    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:31:23.642012 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:23.640857    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:23.796062 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:23.794924    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Oct 31 13:31:23.887625 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:23.887595    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:31:23.921799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:23.921753    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:23.922409 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:23.922379    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:31:24.362806 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:24.362768    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:24.432321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:24.406369    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:24.594837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:24.594146    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:31:24.610036 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:24.606760    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:31:24.827323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:24.792255    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:31:24.827323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:24.812360    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:31:24.845109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:24.842766    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:31:24.846540 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:24.845832    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:31:24.849918 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:24.846882    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:31:25.290421 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:25.290382    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"]
Oct 31 13:31:30.291385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:30.291313    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq" podUID="95f5af14-f5de-4072-aeee-d32d4aaf1bde" containerName="payload-pre-processing" containerID="cri-o://caeecba116583e313c933f6f186956c44b5c5482fc93ab0306a2280f418321e5" gracePeriod=25
Oct 31 13:31:30.660716 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:30.660680    4660 generic.go:356] "Generic (PLEG): container finished" podID="95f5af14-f5de-4072-aeee-d32d4aaf1bde" containerID="caeecba116583e313c933f6f186956c44b5c5482fc93ab0306a2280f418321e5" exitCode=0
Oct 31 13:31:30.660900 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:30.660880    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq" event={"ID":"95f5af14-f5de-4072-aeee-d32d4aaf1bde","Type":"ContainerDied","Data":"caeecba116583e313c933f6f186956c44b5c5482fc93ab0306a2280f418321e5"}
Oct 31 13:31:30.813876 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:30.813843    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"
Oct 31 13:31:30.938358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:30.938012    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95f5af14-f5de-4072-aeee-d32d4aaf1bde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f5af14-f5de-4072-aeee-d32d4aaf1bde-plugins-config-volume\") pod \"95f5af14-f5de-4072-aeee-d32d4aaf1bde\" (UID: \"95f5af14-f5de-4072-aeee-d32d4aaf1bde\") "
Oct 31 13:31:30.938553 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:30.938413    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f5af14-f5de-4072-aeee-d32d4aaf1bde-kube-api-access-pl448\" (UniqueName: \"kubernetes.io/projected/95f5af14-f5de-4072-aeee-d32d4aaf1bde-kube-api-access-pl448\") pod \"95f5af14-f5de-4072-aeee-d32d4aaf1bde\" (UID: \"95f5af14-f5de-4072-aeee-d32d4aaf1bde\") "
Oct 31 13:31:30.938553 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:30.938427    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f5af14-f5de-4072-aeee-d32d4aaf1bde-plugins-config-volume" pod "95f5af14-f5de-4072-aeee-d32d4aaf1bde" (UID: "95f5af14-f5de-4072-aeee-d32d4aaf1bde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:31:30.939708 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:30.939679    4660 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f5af14-f5de-4072-aeee-d32d4aaf1bde-plugins-config-volume\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:30.943374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:30.941413    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f5af14-f5de-4072-aeee-d32d4aaf1bde-kube-api-access-pl448" pod "95f5af14-f5de-4072-aeee-d32d4aaf1bde" (UID: "95f5af14-f5de-4072-aeee-d32d4aaf1bde"). InnerVolumeSpecName "kube-api-access-pl448". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:31:31.042204 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:31.042125    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl448\" (UniqueName: \"kubernetes.io/projected/95f5af14-f5de-4072-aeee-d32d4aaf1bde-kube-api-access-pl448\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:31.675803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:31.675270    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"
Oct 31 13:31:31.675803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:31.675261    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq" event={"ID":"95f5af14-f5de-4072-aeee-d32d4aaf1bde","Type":"ContainerDied","Data":"c3ef3acae9188b56d918d209ce17fc74e10fb4407011d2e12e29a337db034bb0"}
Oct 31 13:31:31.675803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:31.675536    4660 scope.go:122] "RemoveContainer" containerID="caeecba116583e313c933f6f186956c44b5c5482fc93ab0306a2280f418321e5"
Oct 31 13:31:31.747425 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:31.746843    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"]
Oct 31 13:31:31.762314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:31.758396    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8439b78a-7496664b99-2krxq"]
Oct 31 13:31:32.531989 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:32.531950    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95f5af14-f5de-4072-aeee-d32d4aaf1bde" path="/var/lib/kubelet/pods/95f5af14-f5de-4072-aeee-d32d4aaf1bde/volumes"
Oct 31 13:31:33.789315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:33.787080    4660 generic.go:356] "Generic (PLEG): container finished" podID="231f47ca-89fc-4370-853f-b0d788524589" containerID="c80a37c3ea4fd0651136a75d1f1229c4b6dbceaf62cc2f16af8c1c03725046e6" exitCode=0
Oct 31 13:31:33.789315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:33.787362    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d" event={"ID":"231f47ca-89fc-4370-853f-b0d788524589","Type":"ContainerDied","Data":"c80a37c3ea4fd0651136a75d1f1229c4b6dbceaf62cc2f16af8c1c03725046e6"}
Oct 31 13:31:33.789315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:33.787924    4660 scope.go:122] "RemoveContainer" containerID="c80a37c3ea4fd0651136a75d1f1229c4b6dbceaf62cc2f16af8c1c03725046e6"
Oct 31 13:31:33.796581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:33.796542    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:31:33.796805 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:33.796786    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:31:33.796957 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:33.796939    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:31:34.071995 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:34.071959    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"]
Oct 31 13:31:34.799266 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:34.799223    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d" event={"ID":"231f47ca-89fc-4370-853f-b0d788524589","Type":"ContainerStarted","Data":"79eeee4daff267574432fbe251be3d76ab69b05993a3a4183e5d671b6108b72b"}
Oct 31 13:31:34.799783 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:34.799355    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d" podUID="231f47ca-89fc-4370-853f-b0d788524589" containerName="istio-proxy" containerID="cri-o://79eeee4daff267574432fbe251be3d76ab69b05993a3a4183e5d671b6108b72b" gracePeriod=30
Oct 31 13:31:37.402909 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:37.402875    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:31:40.740074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.739623    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:31:40.763106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.761931    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-credential-socket\") pod \"231f47ca-89fc-4370-853f-b0d788524589\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") "
Oct 31 13:31:40.763106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.762017    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-istio-data\") pod \"231f47ca-89fc-4370-853f-b0d788524589\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") "
Oct 31 13:31:40.763106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.762051    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-workload-socket\") pod \"231f47ca-89fc-4370-853f-b0d788524589\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") "
Oct 31 13:31:40.763106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.762112    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231f47ca-89fc-4370-853f-b0d788524589-kube-api-access-hfmmz\" (UniqueName: \"kubernetes.io/projected/231f47ca-89fc-4370-853f-b0d788524589-kube-api-access-hfmmz\") pod \"231f47ca-89fc-4370-853f-b0d788524589\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") "
Oct 31 13:31:40.763106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.762223    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-istio-envoy\") pod \"231f47ca-89fc-4370-853f-b0d788524589\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") "
Oct 31 13:31:40.763106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.762264    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-workload-certs\") pod \"231f47ca-89fc-4370-853f-b0d788524589\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") "
Oct 31 13:31:40.763106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.762382    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231f47ca-89fc-4370-853f-b0d788524589-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231f47ca-89fc-4370-853f-b0d788524589-istiod-ca-cert\") pod \"231f47ca-89fc-4370-853f-b0d788524589\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") "
Oct 31 13:31:40.763106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.762558    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231f47ca-89fc-4370-853f-b0d788524589-istio-token\" (UniqueName: \"kubernetes.io/projected/231f47ca-89fc-4370-853f-b0d788524589-istio-token\") pod \"231f47ca-89fc-4370-853f-b0d788524589\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") "
Oct 31 13:31:40.763106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.762658    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/231f47ca-89fc-4370-853f-b0d788524589-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231f47ca-89fc-4370-853f-b0d788524589-istio-podinfo\") pod \"231f47ca-89fc-4370-853f-b0d788524589\" (UID: \"231f47ca-89fc-4370-853f-b0d788524589\") "
Oct 31 13:31:40.766496 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.765375    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231f47ca-89fc-4370-853f-b0d788524589-istiod-ca-cert" pod "231f47ca-89fc-4370-853f-b0d788524589" (UID: "231f47ca-89fc-4370-853f-b0d788524589"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:31:40.766496 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.765668    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-workload-socket" pod "231f47ca-89fc-4370-853f-b0d788524589" (UID: "231f47ca-89fc-4370-853f-b0d788524589"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:31:40.766496 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.765987    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-workload-certs" pod "231f47ca-89fc-4370-853f-b0d788524589" (UID: "231f47ca-89fc-4370-853f-b0d788524589"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:31:40.768069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.767994    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:40.768069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.768030    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:40.768069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.768044    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231f47ca-89fc-4370-853f-b0d788524589-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:40.769712 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.769679    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-istio-data" pod "231f47ca-89fc-4370-853f-b0d788524589" (UID: "231f47ca-89fc-4370-853f-b0d788524589"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:31:40.771421 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.771367    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-credential-socket" pod "231f47ca-89fc-4370-853f-b0d788524589" (UID: "231f47ca-89fc-4370-853f-b0d788524589"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:31:40.776514 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.776077    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/231f47ca-89fc-4370-853f-b0d788524589-istio-podinfo" pod "231f47ca-89fc-4370-853f-b0d788524589" (UID: "231f47ca-89fc-4370-853f-b0d788524589"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:31:40.776837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.776734    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-istio-envoy" pod "231f47ca-89fc-4370-853f-b0d788524589" (UID: "231f47ca-89fc-4370-853f-b0d788524589"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:31:40.787506 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.786514    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231f47ca-89fc-4370-853f-b0d788524589-istio-token" pod "231f47ca-89fc-4370-853f-b0d788524589" (UID: "231f47ca-89fc-4370-853f-b0d788524589"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:31:40.787506 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.786876    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231f47ca-89fc-4370-853f-b0d788524589-kube-api-access-hfmmz" pod "231f47ca-89fc-4370-853f-b0d788524589" (UID: "231f47ca-89fc-4370-853f-b0d788524589"). InnerVolumeSpecName "kube-api-access-hfmmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:31:40.872324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.872252    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231f47ca-89fc-4370-853f-b0d788524589-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:40.872324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.872280    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:40.872324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.872305    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:40.872324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.872317    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfmmz\" (UniqueName: \"kubernetes.io/projected/231f47ca-89fc-4370-853f-b0d788524589-kube-api-access-hfmmz\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:40.872324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.872327    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231f47ca-89fc-4370-853f-b0d788524589-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:40.872324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.872337    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231f47ca-89fc-4370-853f-b0d788524589-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:40.911503 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.911400    4660 generic.go:356] "Generic (PLEG): container finished" podID="231f47ca-89fc-4370-853f-b0d788524589" containerID="79eeee4daff267574432fbe251be3d76ab69b05993a3a4183e5d671b6108b72b" exitCode=0
Oct 31 13:31:40.911503 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.911463    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"
Oct 31 13:31:40.911736 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.911616    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d" event={"ID":"231f47ca-89fc-4370-853f-b0d788524589","Type":"ContainerDied","Data":"79eeee4daff267574432fbe251be3d76ab69b05993a3a4183e5d671b6108b72b"}
Oct 31 13:31:40.911736 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.911657    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d" event={"ID":"231f47ca-89fc-4370-853f-b0d788524589","Type":"ContainerDied","Data":"4495691bbb86572054435bfe5b895e64ae5109a738eafe36e384104b6414a100"}
Oct 31 13:31:40.911736 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.911683    4660 scope.go:122] "RemoveContainer" containerID="79eeee4daff267574432fbe251be3d76ab69b05993a3a4183e5d671b6108b72b"
Oct 31 13:31:40.932740 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.931808    4660 scope.go:122] "RemoveContainer" containerID="c80a37c3ea4fd0651136a75d1f1229c4b6dbceaf62cc2f16af8c1c03725046e6"
Oct 31 13:31:40.942535 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.942509    4660 scope.go:122] "RemoveContainer" containerID="79eeee4daff267574432fbe251be3d76ab69b05993a3a4183e5d671b6108b72b"
Oct 31 13:31:40.942839 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:31:40.942817    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79eeee4daff267574432fbe251be3d76ab69b05993a3a4183e5d671b6108b72b\": container with ID starting with 79eeee4daff267574432fbe251be3d76ab69b05993a3a4183e5d671b6108b72b not found: ID does not exist" containerID="79eeee4daff267574432fbe251be3d76ab69b05993a3a4183e5d671b6108b72b"
Oct 31 13:31:40.942912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.942849    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79eeee4daff267574432fbe251be3d76ab69b05993a3a4183e5d671b6108b72b"} err="failed to get container status \"79eeee4daff267574432fbe251be3d76ab69b05993a3a4183e5d671b6108b72b\": rpc error: code = NotFound desc = could not find container \"79eeee4daff267574432fbe251be3d76ab69b05993a3a4183e5d671b6108b72b\": container with ID starting with 79eeee4daff267574432fbe251be3d76ab69b05993a3a4183e5d671b6108b72b not found: ID does not exist"
Oct 31 13:31:40.942912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.942866    4660 scope.go:122] "RemoveContainer" containerID="c80a37c3ea4fd0651136a75d1f1229c4b6dbceaf62cc2f16af8c1c03725046e6"
Oct 31 13:31:40.943204 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:31:40.943186    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80a37c3ea4fd0651136a75d1f1229c4b6dbceaf62cc2f16af8c1c03725046e6\": container with ID starting with c80a37c3ea4fd0651136a75d1f1229c4b6dbceaf62cc2f16af8c1c03725046e6 not found: ID does not exist" containerID="c80a37c3ea4fd0651136a75d1f1229c4b6dbceaf62cc2f16af8c1c03725046e6"
Oct 31 13:31:40.943331 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.943213    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80a37c3ea4fd0651136a75d1f1229c4b6dbceaf62cc2f16af8c1c03725046e6"} err="failed to get container status \"c80a37c3ea4fd0651136a75d1f1229c4b6dbceaf62cc2f16af8c1c03725046e6\": rpc error: code = NotFound desc = could not find container \"c80a37c3ea4fd0651136a75d1f1229c4b6dbceaf62cc2f16af8c1c03725046e6\": container with ID starting with c80a37c3ea4fd0651136a75d1f1229c4b6dbceaf62cc2f16af8c1c03725046e6 not found: ID does not exist"
Oct 31 13:31:40.951523 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.951497    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"]
Oct 31 13:31:40.958555 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:40.958510    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8439b78a-openshift-default-5789ddb9c-bjf9d"]
Oct 31 13:31:42.533665 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:31:42.533621    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231f47ca-89fc-4370-853f-b0d788524589" path="/var/lib/kubelet/pods/231f47ca-89fc-4370-853f-b0d788524589/volumes"
Oct 31 13:31:50.740978 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:31:50.740937    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:32:00.086317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:00.086157    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"]
Oct 31 13:32:00.086915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:00.086511    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh" podUID="9926d7e7-5939-4e23-85b0-28682993eab9" containerName="istio-proxy" containerID="cri-o://532133cb4dadcaa80a18eedf5ac336b59ce2d6e2dec5463671c052979d9478f8" gracePeriod=30
Oct 31 13:32:01.212515 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:01.212468    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:01.337436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:01.337392    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:01.379479 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:01.378531    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:01.425105 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:01.425058    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:01.488505 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:01.488398    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:01.813760 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:01.813630    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:02.125074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:02.124688    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:32:02.125074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:02.124769    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc" podUID="344707b1-2bf5-4481-8522-afe6d996c26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:02.362693 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:02.353837    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:02.387200 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:02.368589    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:02.387200 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:02.370050    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:32:02.446338 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:02.388401    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:02.446338 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:02.388932    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:32:02.492766 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:02.492664    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:02.494907 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:02.493701    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:32:02.574655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:02.522250    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:02.574655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:02.574473    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:03.082361 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:03.082315    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:03.172499 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:03.172461    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:32:03.172761 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:03.172734    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9" podUID="ab9e59d3-2ac8-4381-98fa-22d3f25d682b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:03.377399 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:03.377354    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:32:03.395142 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:03.393581    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:03.493031 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:03.492993    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:03.596468 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:03.596432    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:32:03.599731 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:03.599698    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:32:03.681367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:03.677515    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:32:05.481927 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.481835    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:32:05.580055 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.580021    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-workload-socket\") pod \"9926d7e7-5939-4e23-85b0-28682993eab9\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") "
Oct 31 13:32:05.580241 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.580095    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9926d7e7-5939-4e23-85b0-28682993eab9-kube-api-access-7mpdm\" (UniqueName: \"kubernetes.io/projected/9926d7e7-5939-4e23-85b0-28682993eab9-kube-api-access-7mpdm\") pod \"9926d7e7-5939-4e23-85b0-28682993eab9\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") "
Oct 31 13:32:05.580241 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.580214    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-istio-data\") pod \"9926d7e7-5939-4e23-85b0-28682993eab9\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") "
Oct 31 13:32:05.580393 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.580342    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9926d7e7-5939-4e23-85b0-28682993eab9-istio-token\" (UniqueName: \"kubernetes.io/projected/9926d7e7-5939-4e23-85b0-28682993eab9-istio-token\") pod \"9926d7e7-5939-4e23-85b0-28682993eab9\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") "
Oct 31 13:32:05.580393 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.580353    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-workload-socket" pod "9926d7e7-5939-4e23-85b0-28682993eab9" (UID: "9926d7e7-5939-4e23-85b0-28682993eab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:32:05.580570 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.580550    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-istio-data" pod "9926d7e7-5939-4e23-85b0-28682993eab9" (UID: "9926d7e7-5939-4e23-85b0-28682993eab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:32:05.580688 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.580674    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-istio-envoy\") pod \"9926d7e7-5939-4e23-85b0-28682993eab9\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") "
Oct 31 13:32:05.580859 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.580840    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9926d7e7-5939-4e23-85b0-28682993eab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9926d7e7-5939-4e23-85b0-28682993eab9-istiod-ca-cert\") pod \"9926d7e7-5939-4e23-85b0-28682993eab9\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") "
Oct 31 13:32:05.580983 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.580969    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-credential-socket\") pod \"9926d7e7-5939-4e23-85b0-28682993eab9\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") "
Oct 31 13:32:05.581101 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.581087    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-workload-certs\") pod \"9926d7e7-5939-4e23-85b0-28682993eab9\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") "
Oct 31 13:32:05.581305 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.581265    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9926d7e7-5939-4e23-85b0-28682993eab9-istiod-ca-cert" pod "9926d7e7-5939-4e23-85b0-28682993eab9" (UID: "9926d7e7-5939-4e23-85b0-28682993eab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:32:05.581394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.581358    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-workload-certs" pod "9926d7e7-5939-4e23-85b0-28682993eab9" (UID: "9926d7e7-5939-4e23-85b0-28682993eab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:32:05.581526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.581507    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9926d7e7-5939-4e23-85b0-28682993eab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9926d7e7-5939-4e23-85b0-28682993eab9-istio-podinfo\") pod \"9926d7e7-5939-4e23-85b0-28682993eab9\" (UID: \"9926d7e7-5939-4e23-85b0-28682993eab9\") "
Oct 31 13:32:05.581880 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.581538    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-credential-socket" pod "9926d7e7-5939-4e23-85b0-28682993eab9" (UID: "9926d7e7-5939-4e23-85b0-28682993eab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:32:05.582375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.582358    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9926d7e7-5939-4e23-85b0-28682993eab9-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:05.582467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.582459    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:05.582526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.582519    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:05.582579 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.582572    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:05.582636 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.582629    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:05.588382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.583666    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9926d7e7-5939-4e23-85b0-28682993eab9-istio-podinfo" pod "9926d7e7-5939-4e23-85b0-28682993eab9" (UID: "9926d7e7-5939-4e23-85b0-28682993eab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:32:05.588382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.585657    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9926d7e7-5939-4e23-85b0-28682993eab9-istio-token" pod "9926d7e7-5939-4e23-85b0-28682993eab9" (UID: "9926d7e7-5939-4e23-85b0-28682993eab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:32:05.588382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.585695    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9926d7e7-5939-4e23-85b0-28682993eab9-kube-api-access-7mpdm" pod "9926d7e7-5939-4e23-85b0-28682993eab9" (UID: "9926d7e7-5939-4e23-85b0-28682993eab9"). InnerVolumeSpecName "kube-api-access-7mpdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:32:05.588382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.587069    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-istio-envoy" pod "9926d7e7-5939-4e23-85b0-28682993eab9" (UID: "9926d7e7-5939-4e23-85b0-28682993eab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:32:05.651109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.651068    4660 generic.go:356] "Generic (PLEG): container finished" podID="9926d7e7-5939-4e23-85b0-28682993eab9" containerID="532133cb4dadcaa80a18eedf5ac336b59ce2d6e2dec5463671c052979d9478f8" exitCode=0
Oct 31 13:32:05.651350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.651125    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"
Oct 31 13:32:05.651350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.651161    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh" event={"ID":"9926d7e7-5939-4e23-85b0-28682993eab9","Type":"ContainerDied","Data":"532133cb4dadcaa80a18eedf5ac336b59ce2d6e2dec5463671c052979d9478f8"}
Oct 31 13:32:05.651350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.651233    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh" event={"ID":"9926d7e7-5939-4e23-85b0-28682993eab9","Type":"ContainerDied","Data":"e6d34cd4377641cc7893aa5b8662174701b3de1b79ae4bb5c8b164fd43c46361"}
Oct 31 13:32:05.651350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.651267    4660 scope.go:122] "RemoveContainer" containerID="532133cb4dadcaa80a18eedf5ac336b59ce2d6e2dec5463671c052979d9478f8"
Oct 31 13:32:05.663305 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.663275    4660 scope.go:122] "RemoveContainer" containerID="532133cb4dadcaa80a18eedf5ac336b59ce2d6e2dec5463671c052979d9478f8"
Oct 31 13:32:05.663576 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:32:05.663556    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"532133cb4dadcaa80a18eedf5ac336b59ce2d6e2dec5463671c052979d9478f8\": container with ID starting with 532133cb4dadcaa80a18eedf5ac336b59ce2d6e2dec5463671c052979d9478f8 not found: ID does not exist" containerID="532133cb4dadcaa80a18eedf5ac336b59ce2d6e2dec5463671c052979d9478f8"
Oct 31 13:32:05.663651 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.663583    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532133cb4dadcaa80a18eedf5ac336b59ce2d6e2dec5463671c052979d9478f8"} err="failed to get container status \"532133cb4dadcaa80a18eedf5ac336b59ce2d6e2dec5463671c052979d9478f8\": rpc error: code = NotFound desc = could not find container \"532133cb4dadcaa80a18eedf5ac336b59ce2d6e2dec5463671c052979d9478f8\": container with ID starting with 532133cb4dadcaa80a18eedf5ac336b59ce2d6e2dec5463671c052979d9478f8 not found: ID does not exist"
Oct 31 13:32:05.683237 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.683153    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"]
Oct 31 13:32:05.684015 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.683989    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9926d7e7-5939-4e23-85b0-28682993eab9-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:05.684121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.684017    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mpdm\" (UniqueName: \"kubernetes.io/projected/9926d7e7-5939-4e23-85b0-28682993eab9-kube-api-access-7mpdm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:05.684121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.684031    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9926d7e7-5939-4e23-85b0-28682993eab9-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:05.684121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.684044    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9926d7e7-5939-4e23-85b0-28682993eab9-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:05.687466 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:05.687444    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a66ae960-openshift-default-5bdd6f848f-7vwwh"]
Oct 31 13:32:06.533756 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:06.533711    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9926d7e7-5939-4e23-85b0-28682993eab9" path="/var/lib/kubelet/pods/9926d7e7-5939-4e23-85b0-28682993eab9/volumes"
Oct 31 13:32:12.732510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:12.732471    4660 generic.go:356] "Generic (PLEG): container finished" podID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerID="1dce5c0208b70a73c24d94ca1081a813b03e9aef4f6b29f5c62a9a7516f8db71" exitCode=0
Oct 31 13:32:12.732959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:12.732521    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5" event={"ID":"49f9647f-e908-4ede-bb66-683c8f83f0e1","Type":"ContainerDied","Data":"1dce5c0208b70a73c24d94ca1081a813b03e9aef4f6b29f5c62a9a7516f8db71"}
Oct 31 13:32:12.733021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:12.733006    4660 scope.go:122] "RemoveContainer" containerID="1dce5c0208b70a73c24d94ca1081a813b03e9aef4f6b29f5c62a9a7516f8db71"
Oct 31 13:32:12.734816 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:12.734784    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:32:12.734925 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:12.734857    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:32:12.734925 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:12.734889    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:32:13.762882 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:13.762845    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5" event={"ID":"49f9647f-e908-4ede-bb66-683c8f83f0e1","Type":"ContainerStarted","Data":"af4c13e99075145416df3d90c1f74bac76dcd6ddac62fa06cefbf1091b0710b4"}
Oct 31 13:32:14.063655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:14.063568    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:32:14.063916 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:14.063852    4660 patch_prober.go:29] interesting pod/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:32:14.063993 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:14.063941    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5" podUID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 13:32:15.063382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:15.063349    4660 patch_prober.go:29] interesting pod/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:32:15.063787 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:15.063405    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5" podUID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 13:32:16.113320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:16.112574    4660 patch_prober.go:29] interesting pod/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:32:16.113320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:16.112683    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5" podUID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 13:32:17.074317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:17.070123    4660 patch_prober.go:29] interesting pod/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:32:17.074317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:17.070190    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5" podUID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 13:32:18.069330 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:18.066731    4660 patch_prober.go:29] interesting pod/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:32:18.069330 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:18.066794    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5" podUID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 13:32:19.058484 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:19.058448    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:32:19.064664 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:19.064636    4660 patch_prober.go:29] interesting pod/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:32:19.064894 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:19.064870    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5" podUID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 13:32:20.070468 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:20.070431    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:32:20.071672 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:20.071647    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:32:20.745973 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:32:20.745930    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:32:50.753695 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:32:50.753621    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:32:53.169093 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.169048    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"]
Oct 31 13:32:53.170137 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.170095    4660 cpu_manager.go:414] "containerMap: removing container" podUID="231f47ca-89fc-4370-853f-b0d788524589" containerName="istio-proxy"
Oct 31 13:32:53.170137 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.170122    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231f47ca-89fc-4370-853f-b0d788524589" containerName="istio-proxy"
Oct 31 13:32:53.170387 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.170150    4660 cpu_manager.go:414] "containerMap: removing container" podUID="9926d7e7-5939-4e23-85b0-28682993eab9" containerName="istio-proxy"
Oct 31 13:32:53.170387 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.170158    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9926d7e7-5939-4e23-85b0-28682993eab9" containerName="istio-proxy"
Oct 31 13:32:53.170387 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.170177    4660 cpu_manager.go:414] "containerMap: removing container" podUID="95f5af14-f5de-4072-aeee-d32d4aaf1bde" containerName="payload-pre-processing"
Oct 31 13:32:53.170387 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.170185    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f5af14-f5de-4072-aeee-d32d4aaf1bde" containerName="payload-pre-processing"
Oct 31 13:32:53.170387 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.170241    4660 cpu_manager.go:414] "containerMap: removing container" podUID="231f47ca-89fc-4370-853f-b0d788524589" containerName="istio-proxy"
Oct 31 13:32:53.170387 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.170248    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231f47ca-89fc-4370-853f-b0d788524589" containerName="istio-proxy"
Oct 31 13:32:53.170687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.170439    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="231f47ca-89fc-4370-853f-b0d788524589" containerName="istio-proxy"
Oct 31 13:32:53.170687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.170458    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="9926d7e7-5939-4e23-85b0-28682993eab9" containerName="istio-proxy"
Oct 31 13:32:53.170687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.170469    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f5af14-f5de-4072-aeee-d32d4aaf1bde" containerName="payload-pre-processing"
Oct 31 13:32:53.173646 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.173624    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.183514 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.183478    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"]
Oct 31 13:32:53.183899 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.183872    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-79f4fb0f\"/\"default-dockercfg-g4xgj\""
Oct 31 13:32:53.184527 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.183999    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-79f4fb0f\"/\"openshift-service-ca.crt\""
Oct 31 13:32:53.184660 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.184086    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-79f4fb0f\"/\"kube-root-ca.crt\""
Oct 31 13:32:53.184737 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.184136    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-79f4fb0f\"/\"e2e-lifecycle-model-79f4fb0f-kserve-self-signed-certs\""
Oct 31 13:32:53.298118 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.298082    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-dshm\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.298118 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.298137    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a40e94-c560-4f09-b721-533261b4ff25-tls-certs\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.298372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.298173    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqmgw\" (UniqueName: \"kubernetes.io/projected/c1a40e94-c560-4f09-b721-533261b4ff25-kube-api-access-vqmgw\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.298372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.298340    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-tmp-dir\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.298477 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.298457    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-home\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.298602 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.298586    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-model-cache\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.410318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.410195    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-dshm\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.410318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.410280    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a40e94-c560-4f09-b721-533261b4ff25-tls-certs\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.410583 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.410333    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqmgw\" (UniqueName: \"kubernetes.io/projected/c1a40e94-c560-4f09-b721-533261b4ff25-kube-api-access-vqmgw\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.410583 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.410370    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-tmp-dir\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.410583 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.410456    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-home\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.410752 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.410590    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-model-cache\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.427367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.411143    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-model-cache\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.427367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.411710    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-tmp-dir\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.427367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.412272    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-home\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.427367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.423226    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a40e94-c560-4f09-b721-533261b4ff25-tls-certs\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.459314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.458811    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-dshm\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.484319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.471882    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqmgw\" (UniqueName: \"kubernetes.io/projected/c1a40e94-c560-4f09-b721-533261b4ff25-kube-api-access-vqmgw\") pod \"e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") " pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.497348 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.493853    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:32:53.964618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:53.964469    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"]
Oct 31 13:32:53.964992 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:32:53.964966    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1a40e94_c560_4f09_b721_533261b4ff25.slice/crio-83abc515e54f66fb82ee050d57928d47b0227e1c687dbfa26c535907521707a4 WatchSource:0}: Error finding container 83abc515e54f66fb82ee050d57928d47b0227e1c687dbfa26c535907521707a4: Status 404 returned error can't find the container with id 83abc515e54f66fb82ee050d57928d47b0227e1c687dbfa26c535907521707a4
Oct 31 13:32:54.702733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:54.702679    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq" event={"ID":"c1a40e94-c560-4f09-b721-533261b4ff25","Type":"ContainerStarted","Data":"82dade06584e27c60fa6f915cd4a559fa6eff51d147214467bf04661c46f04f0"}
Oct 31 13:32:54.702733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:54.702732    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq" event={"ID":"c1a40e94-c560-4f09-b721-533261b4ff25","Type":"ContainerStarted","Data":"83abc515e54f66fb82ee050d57928d47b0227e1c687dbfa26c535907521707a4"}
Oct 31 13:32:54.734864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:54.734795    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq" podStartSLOduration=1.734772286 podStartE2EDuration="1.734772286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:32:54.724043641 +0000 UTC m=+1774.342702806" watchObservedRunningTime="2026-10-31 13:32:54.734772286 +0000 UTC m=+1774.353431454"
Oct 31 13:32:55.494309 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:32:55.494259    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:33:03.494467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:03.494423    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:33:03.508977 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:03.508948    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:33:03.836612 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:03.836513    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:33:20.739055 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:33:20.738996    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:33:20.814715 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:20.813482    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:33:20.825269 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:20.825235    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:33:20.826834 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:20.826808    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:33:20.862276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:20.862238    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:33:20.874992 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:20.874960    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:33:20.881197 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:20.881166    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:33:43.759018 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.758966    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"]
Oct 31 13:33:43.763314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.760263    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="231f47ca-89fc-4370-853f-b0d788524589" containerName="istio-proxy"
Oct 31 13:33:43.764973 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.764733    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.787502 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.787466    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-33c10a-openshift-default-dockercfg-ptdwg\""
Oct 31 13:33:43.801861 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.801817    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"]
Oct 31 13:33:43.845644 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.845608    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-workload-certs\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.845847 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.845650    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m97pn\" (UniqueName: \"kubernetes.io/projected/e71b679c-5dff-4c19-bef4-a4bf19093b34-kube-api-access-m97pn\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.845920 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.845873    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-podinfo\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.845978 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.845935    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71b679c-5dff-4c19-bef4-a4bf19093b34-istiod-ca-cert\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.846034 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.846000    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-data\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.846101 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.846085    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-credential-socket\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.846179 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.846165    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-token\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.846223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.846207    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-workload-socket\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.846258 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.846229    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-envoy\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.952318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.951534    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-podinfo\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.952318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.951617    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71b679c-5dff-4c19-bef4-a4bf19093b34-istiod-ca-cert\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.952318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.951662    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-data\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.952318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.951737    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-credential-socket\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.952318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.951832    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-token\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.952318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.951868    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-workload-socket\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.952318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.951900    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-envoy\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.952318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.952044    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-workload-certs\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.952318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.952074    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m97pn\" (UniqueName: \"kubernetes.io/projected/e71b679c-5dff-4c19-bef4-a4bf19093b34-kube-api-access-m97pn\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.955449 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.955425    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-podinfo\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.955705 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.955672    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-workload-socket\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.956388 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.956361    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71b679c-5dff-4c19-bef4-a4bf19093b34-istiod-ca-cert\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.959391 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.958677    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-credential-socket\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.959524 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.959500    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-workload-certs\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.959612 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.958912    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-data\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.960030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.959971    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-envoy\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.972423 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.972279    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-token\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:43.977013 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:43.976920    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m97pn\" (UniqueName: \"kubernetes.io/projected/e71b679c-5dff-4c19-bef4-a4bf19093b34-kube-api-access-m97pn\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:44.116815 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:44.116723    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:44.533901 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:44.533223    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"]
Oct 31 13:33:44.615324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:44.610308    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:33:44.615324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:44.610414    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:33:44.615324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:44.610456    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:33:45.276950 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:45.276911    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn" event={"ID":"e71b679c-5dff-4c19-bef4-a4bf19093b34","Type":"ContainerStarted","Data":"5f72848048b7bb19b1b96647657945c5c2563c2767c3bb671e6f5d90a79e7d72"}
Oct 31 13:33:45.277463 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:45.276988    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn" event={"ID":"e71b679c-5dff-4c19-bef4-a4bf19093b34","Type":"ContainerStarted","Data":"d81a0c2360fbb7671a5535ee44e6ab9383b237881e9ec0d20ce6aa47e4274d0f"}
Oct 31 13:33:45.318868 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:45.318804    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn" podStartSLOduration=2.318786229 podStartE2EDuration="2.318786229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:33:45.308903575 +0000 UTC m=+1824.927562740" watchObservedRunningTime="2026-10-31 13:33:45.318786229 +0000 UTC m=+1824.937445478"
Oct 31 13:33:46.117776 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:46.117736    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:46.125927 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:46.125894    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:46.291999 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:46.291950    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:46.292945 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:46.292924    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:33:46.923480 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:46.923440    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"]
Oct 31 13:33:46.923729 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:46.923698    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5" podUID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerName="istio-proxy" containerID="cri-o://af4c13e99075145416df3d90c1f74bac76dcd6ddac62fa06cefbf1091b0710b4" gracePeriod=30
Oct 31 13:33:50.073779 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:50.073740    4660 patch_prober.go:29] interesting pod/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:33:50.074356 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:50.073802    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5" podUID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:33:50.736679 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:33:50.736635    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:33:52.227903 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.227869    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:33:52.275518 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.275477    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-credential-socket\") pod \"49f9647f-e908-4ede-bb66-683c8f83f0e1\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") "
Oct 31 13:33:52.275879 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.275576    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-token\" (UniqueName: \"kubernetes.io/projected/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-token\") pod \"49f9647f-e908-4ede-bb66-683c8f83f0e1\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") "
Oct 31 13:33:52.275879 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.275710    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-podinfo\") pod \"49f9647f-e908-4ede-bb66-683c8f83f0e1\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") "
Oct 31 13:33:52.275879 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.275810    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f9647f-e908-4ede-bb66-683c8f83f0e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49f9647f-e908-4ede-bb66-683c8f83f0e1-istiod-ca-cert\") pod \"49f9647f-e908-4ede-bb66-683c8f83f0e1\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") "
Oct 31 13:33:52.275879 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.275847    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f9647f-e908-4ede-bb66-683c8f83f0e1-kube-api-access-kqr8q\" (UniqueName: \"kubernetes.io/projected/49f9647f-e908-4ede-bb66-683c8f83f0e1-kube-api-access-kqr8q\") pod \"49f9647f-e908-4ede-bb66-683c8f83f0e1\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") "
Oct 31 13:33:52.275879 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.275846    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-credential-socket" pod "49f9647f-e908-4ede-bb66-683c8f83f0e1" (UID: "49f9647f-e908-4ede-bb66-683c8f83f0e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:33:52.276281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.275957    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-envoy\") pod \"49f9647f-e908-4ede-bb66-683c8f83f0e1\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") "
Oct 31 13:33:52.276281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.276014    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-workload-socket\") pod \"49f9647f-e908-4ede-bb66-683c8f83f0e1\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") "
Oct 31 13:33:52.276281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.276069    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-data\") pod \"49f9647f-e908-4ede-bb66-683c8f83f0e1\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") "
Oct 31 13:33:52.276281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.276104    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-workload-certs\") pod \"49f9647f-e908-4ede-bb66-683c8f83f0e1\" (UID: \"49f9647f-e908-4ede-bb66-683c8f83f0e1\") "
Oct 31 13:33:52.276911 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.276883    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:52.277701 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.277111    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-workload-certs" pod "49f9647f-e908-4ede-bb66-683c8f83f0e1" (UID: "49f9647f-e908-4ede-bb66-683c8f83f0e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:33:52.277701 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.277283    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-data" pod "49f9647f-e908-4ede-bb66-683c8f83f0e1" (UID: "49f9647f-e908-4ede-bb66-683c8f83f0e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:33:52.280514 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.280478    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-workload-socket" pod "49f9647f-e908-4ede-bb66-683c8f83f0e1" (UID: "49f9647f-e908-4ede-bb66-683c8f83f0e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:33:52.280731 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.280710    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f9647f-e908-4ede-bb66-683c8f83f0e1-istiod-ca-cert" pod "49f9647f-e908-4ede-bb66-683c8f83f0e1" (UID: "49f9647f-e908-4ede-bb66-683c8f83f0e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:33:52.282287 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.282253    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-token" pod "49f9647f-e908-4ede-bb66-683c8f83f0e1" (UID: "49f9647f-e908-4ede-bb66-683c8f83f0e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:33:52.285799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.285760    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f9647f-e908-4ede-bb66-683c8f83f0e1-kube-api-access-kqr8q" pod "49f9647f-e908-4ede-bb66-683c8f83f0e1" (UID: "49f9647f-e908-4ede-bb66-683c8f83f0e1"). InnerVolumeSpecName "kube-api-access-kqr8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:33:52.288437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.288405    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-podinfo" pod "49f9647f-e908-4ede-bb66-683c8f83f0e1" (UID: "49f9647f-e908-4ede-bb66-683c8f83f0e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:33:52.307844 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.307803    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-envoy" pod "49f9647f-e908-4ede-bb66-683c8f83f0e1" (UID: "49f9647f-e908-4ede-bb66-683c8f83f0e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:33:52.360735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.360437    4660 generic.go:356] "Generic (PLEG): container finished" podID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerID="af4c13e99075145416df3d90c1f74bac76dcd6ddac62fa06cefbf1091b0710b4" exitCode=0
Oct 31 13:33:52.360735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.360509    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5" event={"ID":"49f9647f-e908-4ede-bb66-683c8f83f0e1","Type":"ContainerDied","Data":"af4c13e99075145416df3d90c1f74bac76dcd6ddac62fa06cefbf1091b0710b4"}
Oct 31 13:33:52.360735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.360525    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"
Oct 31 13:33:52.360735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.360589    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5" event={"ID":"49f9647f-e908-4ede-bb66-683c8f83f0e1","Type":"ContainerDied","Data":"c2ab8030693ef48adbcf0b19a5348fd773a4aa0bc326b17a3cb47d885d42f4ce"}
Oct 31 13:33:52.360735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.360620    4660 scope.go:122] "RemoveContainer" containerID="af4c13e99075145416df3d90c1f74bac76dcd6ddac62fa06cefbf1091b0710b4"
Oct 31 13:33:52.376025 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.376003    4660 scope.go:122] "RemoveContainer" containerID="1dce5c0208b70a73c24d94ca1081a813b03e9aef4f6b29f5c62a9a7516f8db71"
Oct 31 13:33:52.378591 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.378535    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49f9647f-e908-4ede-bb66-683c8f83f0e1-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:52.378591 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.378564    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqr8q\" (UniqueName: \"kubernetes.io/projected/49f9647f-e908-4ede-bb66-683c8f83f0e1-kube-api-access-kqr8q\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:52.378591 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.378581    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:52.378773 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.378595    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:52.378773 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.378609    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:52.378773 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.378623    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49f9647f-e908-4ede-bb66-683c8f83f0e1-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:52.378773 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.378636    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:52.378773 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.378653    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49f9647f-e908-4ede-bb66-683c8f83f0e1-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:52.389753 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.389729    4660 scope.go:122] "RemoveContainer" containerID="af4c13e99075145416df3d90c1f74bac76dcd6ddac62fa06cefbf1091b0710b4"
Oct 31 13:33:52.390146 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:33:52.390128    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af4c13e99075145416df3d90c1f74bac76dcd6ddac62fa06cefbf1091b0710b4\": container with ID starting with af4c13e99075145416df3d90c1f74bac76dcd6ddac62fa06cefbf1091b0710b4 not found: ID does not exist" containerID="af4c13e99075145416df3d90c1f74bac76dcd6ddac62fa06cefbf1091b0710b4"
Oct 31 13:33:52.390196 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.390157    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af4c13e99075145416df3d90c1f74bac76dcd6ddac62fa06cefbf1091b0710b4"} err="failed to get container status \"af4c13e99075145416df3d90c1f74bac76dcd6ddac62fa06cefbf1091b0710b4\": rpc error: code = NotFound desc = could not find container \"af4c13e99075145416df3d90c1f74bac76dcd6ddac62fa06cefbf1091b0710b4\": container with ID starting with af4c13e99075145416df3d90c1f74bac76dcd6ddac62fa06cefbf1091b0710b4 not found: ID does not exist"
Oct 31 13:33:52.390196 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.390176    4660 scope.go:122] "RemoveContainer" containerID="1dce5c0208b70a73c24d94ca1081a813b03e9aef4f6b29f5c62a9a7516f8db71"
Oct 31 13:33:52.390521 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:33:52.390496    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dce5c0208b70a73c24d94ca1081a813b03e9aef4f6b29f5c62a9a7516f8db71\": container with ID starting with 1dce5c0208b70a73c24d94ca1081a813b03e9aef4f6b29f5c62a9a7516f8db71 not found: ID does not exist" containerID="1dce5c0208b70a73c24d94ca1081a813b03e9aef4f6b29f5c62a9a7516f8db71"
Oct 31 13:33:52.390613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.390529    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dce5c0208b70a73c24d94ca1081a813b03e9aef4f6b29f5c62a9a7516f8db71"} err="failed to get container status \"1dce5c0208b70a73c24d94ca1081a813b03e9aef4f6b29f5c62a9a7516f8db71\": rpc error: code = NotFound desc = could not find container \"1dce5c0208b70a73c24d94ca1081a813b03e9aef4f6b29f5c62a9a7516f8db71\": container with ID starting with 1dce5c0208b70a73c24d94ca1081a813b03e9aef4f6b29f5c62a9a7516f8db71 not found: ID does not exist"
Oct 31 13:33:52.402032 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.401993    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"]
Oct 31 13:33:52.413354 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.411005    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a8c1600c-openshift-default-74b4bf4f85-glvg5"]
Oct 31 13:33:52.530218 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:52.530136    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f9647f-e908-4ede-bb66-683c8f83f0e1" path="/var/lib/kubelet/pods/49f9647f-e908-4ede-bb66-683c8f83f0e1/volumes"
Oct 31 13:33:53.729388 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.729346    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"]
Oct 31 13:33:53.730464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.730437    4660 cpu_manager.go:414] "containerMap: removing container" podUID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerName="istio-proxy"
Oct 31 13:33:53.730615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.730465    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerName="istio-proxy"
Oct 31 13:33:53.730615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.730505    4660 cpu_manager.go:414] "containerMap: removing container" podUID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerName="istio-proxy"
Oct 31 13:33:53.730615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.730514    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerName="istio-proxy"
Oct 31 13:33:53.730792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.730672    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerName="istio-proxy"
Oct 31 13:33:53.737506 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.736660    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"]
Oct 31 13:33:53.737506 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.736774    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:33:53.741951 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.741926    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-466002-serving-cert\""
Oct 31 13:33:53.800076 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.800037    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d603cf6-4cc4-4159-a02f-1382522bf109-metrics-certs\") pod \"maas-api-e2e-api-a-466002-7dbdd75659-6jfcd\" (UID: \"3d603cf6-4cc4-4159-a02f-1382522bf109\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:33:53.800254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.800081    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4gm\" (UniqueName: \"kubernetes.io/projected/3d603cf6-4cc4-4159-a02f-1382522bf109-kube-api-access-vn4gm\") pod \"maas-api-e2e-api-a-466002-7dbdd75659-6jfcd\" (UID: \"3d603cf6-4cc4-4159-a02f-1382522bf109\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:33:53.800254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.800190    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d603cf6-4cc4-4159-a02f-1382522bf109-openshift-service-ca\") pod \"maas-api-e2e-api-a-466002-7dbdd75659-6jfcd\" (UID: \"3d603cf6-4cc4-4159-a02f-1382522bf109\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:33:53.800387 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.800364    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d603cf6-4cc4-4159-a02f-1382522bf109-maas-api-tls\") pod \"maas-api-e2e-api-a-466002-7dbdd75659-6jfcd\" (UID: \"3d603cf6-4cc4-4159-a02f-1382522bf109\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:33:53.902083 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.902044    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d603cf6-4cc4-4159-a02f-1382522bf109-metrics-certs\") pod \"maas-api-e2e-api-a-466002-7dbdd75659-6jfcd\" (UID: \"3d603cf6-4cc4-4159-a02f-1382522bf109\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:33:53.902083 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.902091    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4gm\" (UniqueName: \"kubernetes.io/projected/3d603cf6-4cc4-4159-a02f-1382522bf109-kube-api-access-vn4gm\") pod \"maas-api-e2e-api-a-466002-7dbdd75659-6jfcd\" (UID: \"3d603cf6-4cc4-4159-a02f-1382522bf109\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:33:53.902334 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.902147    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d603cf6-4cc4-4159-a02f-1382522bf109-openshift-service-ca\") pod \"maas-api-e2e-api-a-466002-7dbdd75659-6jfcd\" (UID: \"3d603cf6-4cc4-4159-a02f-1382522bf109\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:33:53.902334 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.902227    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d603cf6-4cc4-4159-a02f-1382522bf109-maas-api-tls\") pod \"maas-api-e2e-api-a-466002-7dbdd75659-6jfcd\" (UID: \"3d603cf6-4cc4-4159-a02f-1382522bf109\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:33:53.903268 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.903241    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d603cf6-4cc4-4159-a02f-1382522bf109-openshift-service-ca\") pod \"maas-api-e2e-api-a-466002-7dbdd75659-6jfcd\" (UID: \"3d603cf6-4cc4-4159-a02f-1382522bf109\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:33:53.904959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.904939    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d603cf6-4cc4-4159-a02f-1382522bf109-maas-api-tls\") pod \"maas-api-e2e-api-a-466002-7dbdd75659-6jfcd\" (UID: \"3d603cf6-4cc4-4159-a02f-1382522bf109\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:33:53.905288 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.905266    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d603cf6-4cc4-4159-a02f-1382522bf109-metrics-certs\") pod \"maas-api-e2e-api-a-466002-7dbdd75659-6jfcd\" (UID: \"3d603cf6-4cc4-4159-a02f-1382522bf109\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:33:53.936702 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:53.936666    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4gm\" (UniqueName: \"kubernetes.io/projected/3d603cf6-4cc4-4159-a02f-1382522bf109-kube-api-access-vn4gm\") pod \"maas-api-e2e-api-a-466002-7dbdd75659-6jfcd\" (UID: \"3d603cf6-4cc4-4159-a02f-1382522bf109\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:33:54.061732 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:54.061642    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:33:54.468317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:54.464176    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"]
Oct 31 13:33:55.424447 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:55.424409    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd" event={"ID":"3d603cf6-4cc4-4159-a02f-1382522bf109","Type":"ContainerStarted","Data":"4ddf79e66d9ad6546925dfdaa706507668b9b1fd32d53229a051861307bed516"}
Oct 31 13:33:55.424953 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:55.424506    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd" event={"ID":"3d603cf6-4cc4-4159-a02f-1382522bf109","Type":"ContainerStarted","Data":"dd1d7d237fe3835fa9384d82fff55dcbf9f6997df47f7d2e927d85b3fbdee270"}
Oct 31 13:33:55.424953 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:55.424634    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:33:55.452809 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:33:55.452755    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd" podStartSLOduration=1.95906199 podStartE2EDuration="2.452740938s" totalImagesPullingTime="493.678948ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:33:55.446551641 +0000 UTC m=+1835.065210809" watchObservedRunningTime="2026-10-31 13:33:55.452740938 +0000 UTC m=+1835.071400101"
Oct 31 13:34:01.444218 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:01.444135    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:34:10.104529 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.104488    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"]
Oct 31 13:34:10.105355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.105334    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f9647f-e908-4ede-bb66-683c8f83f0e1" containerName="istio-proxy"
Oct 31 13:34:10.109261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.109237    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:34:10.130053 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.127127    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-75c956-serving-cert\""
Oct 31 13:34:10.132068 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.132020    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"]
Oct 31 13:34:10.192716 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.192669    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53bbd017-4f45-4cfb-9d13-e50b750a2590-maas-api-tls\") pod \"maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt\" (UID: \"53bbd017-4f45-4cfb-9d13-e50b750a2590\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:34:10.192900 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.192833    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bbd017-4f45-4cfb-9d13-e50b750a2590-openshift-service-ca\") pod \"maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt\" (UID: \"53bbd017-4f45-4cfb-9d13-e50b750a2590\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:34:10.192900 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.192866    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jl6n\" (UniqueName: \"kubernetes.io/projected/53bbd017-4f45-4cfb-9d13-e50b750a2590-kube-api-access-2jl6n\") pod \"maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt\" (UID: \"53bbd017-4f45-4cfb-9d13-e50b750a2590\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:34:10.193056 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.193022    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53bbd017-4f45-4cfb-9d13-e50b750a2590-metrics-certs\") pod \"maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt\" (UID: \"53bbd017-4f45-4cfb-9d13-e50b750a2590\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:34:10.294723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.294666    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bbd017-4f45-4cfb-9d13-e50b750a2590-openshift-service-ca\") pod \"maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt\" (UID: \"53bbd017-4f45-4cfb-9d13-e50b750a2590\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:34:10.294921 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.294731    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jl6n\" (UniqueName: \"kubernetes.io/projected/53bbd017-4f45-4cfb-9d13-e50b750a2590-kube-api-access-2jl6n\") pod \"maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt\" (UID: \"53bbd017-4f45-4cfb-9d13-e50b750a2590\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:34:10.294921 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.294822    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53bbd017-4f45-4cfb-9d13-e50b750a2590-metrics-certs\") pod \"maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt\" (UID: \"53bbd017-4f45-4cfb-9d13-e50b750a2590\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:34:10.295054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.295032    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53bbd017-4f45-4cfb-9d13-e50b750a2590-maas-api-tls\") pod \"maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt\" (UID: \"53bbd017-4f45-4cfb-9d13-e50b750a2590\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:34:10.296768 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.296736    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bbd017-4f45-4cfb-9d13-e50b750a2590-openshift-service-ca\") pod \"maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt\" (UID: \"53bbd017-4f45-4cfb-9d13-e50b750a2590\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:34:10.316710 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.316665    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53bbd017-4f45-4cfb-9d13-e50b750a2590-metrics-certs\") pod \"maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt\" (UID: \"53bbd017-4f45-4cfb-9d13-e50b750a2590\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:34:10.317450 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.317425    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53bbd017-4f45-4cfb-9d13-e50b750a2590-maas-api-tls\") pod \"maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt\" (UID: \"53bbd017-4f45-4cfb-9d13-e50b750a2590\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:34:10.319260 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.319231    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jl6n\" (UniqueName: \"kubernetes.io/projected/53bbd017-4f45-4cfb-9d13-e50b750a2590-kube-api-access-2jl6n\") pod \"maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt\" (UID: \"53bbd017-4f45-4cfb-9d13-e50b750a2590\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:34:10.445350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.443654    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:34:10.966315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:10.962590    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"]
Oct 31 13:34:10.975077 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:34:10.975044    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bbd017_4f45_4cfb_9d13_e50b750a2590.slice/crio-4d54a3823c7b2217e3795d43e3c483c4e24a229daa7598e86815569107fb68d4 WatchSource:0}: Error finding container 4d54a3823c7b2217e3795d43e3c483c4e24a229daa7598e86815569107fb68d4: Status 404 returned error can't find the container with id 4d54a3823c7b2217e3795d43e3c483c4e24a229daa7598e86815569107fb68d4
Oct 31 13:34:11.611424 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:11.611385    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt" event={"ID":"53bbd017-4f45-4cfb-9d13-e50b750a2590","Type":"ContainerStarted","Data":"aa041fcaf49c4f909304b9fa18fa6322caef715a8303ef79993e8166b8eb2525"}
Oct 31 13:34:11.611847 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:11.611448    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt" event={"ID":"53bbd017-4f45-4cfb-9d13-e50b750a2590","Type":"ContainerStarted","Data":"4d54a3823c7b2217e3795d43e3c483c4e24a229daa7598e86815569107fb68d4"}
Oct 31 13:34:11.611847 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:11.611470    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:34:11.638834 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:11.638728    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt" podStartSLOduration=1.242019944 podStartE2EDuration="1.638705034s" totalImagesPullingTime="396.68509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:11.632685059 +0000 UTC m=+1851.251344225" watchObservedRunningTime="2026-10-31 13:34:11.638705034 +0000 UTC m=+1851.257364200"
Oct 31 13:34:13.598050 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:13.598020    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11110-2b50a4"]
Oct 31 13:34:13.605457 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:13.605166    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11110-2b50a4"
Oct 31 13:34:13.615054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:13.615022    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11110-2b50a4"]
Oct 31 13:34:13.615667 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:13.615637    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-whw9x\""
Oct 31 13:34:13.625004 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:13.624970    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhcdn\" (UniqueName: \"kubernetes.io/projected/08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9-kube-api-access-mhcdn\") pod \"test-curl-11110-2b50a4\" (UID: \"08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9\") " pod="opendatahub/test-curl-11110-2b50a4"
Oct 31 13:34:13.726754 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:13.726719    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcdn\" (UniqueName: \"kubernetes.io/projected/08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9-kube-api-access-mhcdn\") pod \"test-curl-11110-2b50a4\" (UID: \"08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9\") " pod="opendatahub/test-curl-11110-2b50a4"
Oct 31 13:34:13.742518 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:13.742481    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhcdn\" (UniqueName: \"kubernetes.io/projected/08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9-kube-api-access-mhcdn\") pod \"test-curl-11110-2b50a4\" (UID: \"08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9\") " pod="opendatahub/test-curl-11110-2b50a4"
Oct 31 13:34:13.933768 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:13.933725    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11110-2b50a4"
Oct 31 13:34:14.316471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:14.316427    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11110-2b50a4"]
Oct 31 13:34:14.316773 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:34:14.316745    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08ecf4a9_dd84_4774_8ff7_7b14a3d79ff9.slice/crio-e375ee1c889412a5777f524c3751692e99d0d7daa170c07f17716ec202528e05 WatchSource:0}: Error finding container e375ee1c889412a5777f524c3751692e99d0d7daa170c07f17716ec202528e05: Status 404 returned error can't find the container with id e375ee1c889412a5777f524c3751692e99d0d7daa170c07f17716ec202528e05
Oct 31 13:34:14.656733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:14.656690    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11110-2b50a4" event={"ID":"08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9","Type":"ContainerStarted","Data":"c9367d536b3aa16c4137a2961b9fcf0e48d1e3df917fab2d1893e54219ef012e"}
Oct 31 13:34:14.656733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:14.656746    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11110-2b50a4" event={"ID":"08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9","Type":"ContainerStarted","Data":"e375ee1c889412a5777f524c3751692e99d0d7daa170c07f17716ec202528e05"}
Oct 31 13:34:14.689650 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:14.684346    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11110-2b50a4" podStartSLOduration=1.684324409 podStartE2EDuration="1.684324409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:14.673803592 +0000 UTC m=+1854.292462757" watchObservedRunningTime="2026-10-31 13:34:14.684324409 +0000 UTC m=+1854.302983576"
Oct 31 13:34:15.484709 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:15.484624    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11110-2b50a4"]
Oct 31 13:34:15.500330 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:15.495540    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11110-2b50a4"]
Oct 31 13:34:16.685565 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:16.685515    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11110-2b50a4" podUID="08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9" containerName="test-curl-11110-2b50a4" containerID="cri-o://c9367d536b3aa16c4137a2961b9fcf0e48d1e3df917fab2d1893e54219ef012e" gracePeriod=2
Oct 31 13:34:17.628059 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:17.628029    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:34:19.018217 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:19.018188    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11110-2b50a4"
Oct 31 13:34:19.167653 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:19.167622    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9-kube-api-access-mhcdn\" (UniqueName: \"kubernetes.io/projected/08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9-kube-api-access-mhcdn\") pod \"08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9\" (UID: \"08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9\") "
Oct 31 13:34:19.170153 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:19.170114    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9-kube-api-access-mhcdn" pod "08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9" (UID: "08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9"). InnerVolumeSpecName "kube-api-access-mhcdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:34:19.283372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:19.270905    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhcdn\" (UniqueName: \"kubernetes.io/projected/08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9-kube-api-access-mhcdn\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:34:19.768080 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:19.765404    4660 generic.go:356] "Generic (PLEG): container finished" podID="08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9" containerID="c9367d536b3aa16c4137a2961b9fcf0e48d1e3df917fab2d1893e54219ef012e" exitCode=137
Oct 31 13:34:19.768080 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:19.765799    4660 scope.go:122] "RemoveContainer" containerID="c9367d536b3aa16c4137a2961b9fcf0e48d1e3df917fab2d1893e54219ef012e"
Oct 31 13:34:19.768080 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:19.766408    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11110-2b50a4"
Oct 31 13:34:19.818354 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:19.811640    4660 scope.go:122] "RemoveContainer" containerID="c9367d536b3aa16c4137a2961b9fcf0e48d1e3df917fab2d1893e54219ef012e"
Oct 31 13:34:19.818354 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:34:19.812043    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9367d536b3aa16c4137a2961b9fcf0e48d1e3df917fab2d1893e54219ef012e\": container with ID starting with c9367d536b3aa16c4137a2961b9fcf0e48d1e3df917fab2d1893e54219ef012e not found: ID does not exist" containerID="c9367d536b3aa16c4137a2961b9fcf0e48d1e3df917fab2d1893e54219ef012e"
Oct 31 13:34:19.818354 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:19.812087    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9367d536b3aa16c4137a2961b9fcf0e48d1e3df917fab2d1893e54219ef012e"} err="failed to get container status \"c9367d536b3aa16c4137a2961b9fcf0e48d1e3df917fab2d1893e54219ef012e\": rpc error: code = NotFound desc = could not find container \"c9367d536b3aa16c4137a2961b9fcf0e48d1e3df917fab2d1893e54219ef012e\": container with ID starting with c9367d536b3aa16c4137a2961b9fcf0e48d1e3df917fab2d1893e54219ef012e not found: ID does not exist"
Oct 31 13:34:19.826225 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:34:19.819548    4660 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11110-2b50a4\" is forbidden: User \"system:node:ip-10-0-137-77.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-77.us-east-2.compute.internal' and this object" podUID="08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9" pod="opendatahub/test-curl-11110-2b50a4"
Oct 31 13:34:20.529374 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:34:20.529333    4660 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11110-2b50a4\" is forbidden: User \"system:node:ip-10-0-137-77.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-77.us-east-2.compute.internal' and this object" podUID="08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9" pod="opendatahub/test-curl-11110-2b50a4"
Oct 31 13:34:20.533947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:20.533912    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9" path="/var/lib/kubelet/pods/08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9/volumes"
Oct 31 13:34:20.742707 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:34:20.742672    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:34:26.692230 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.689514    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"]
Oct 31 13:34:26.692230 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.690555    4660 cpu_manager.go:414] "containerMap: removing container" podUID="08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9" containerName="test-curl-11110-2b50a4"
Oct 31 13:34:26.692230 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.690573    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9" containerName="test-curl-11110-2b50a4"
Oct 31 13:34:26.692230 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.690783    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ecf4a9-dd84-4774-8ff7-7b14a3d79ff9" containerName="test-curl-11110-2b50a4"
Oct 31 13:34:26.696706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.696683    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:34:26.703710 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.703567    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-752869-serving-cert\""
Oct 31 13:34:26.716215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.716037    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"]
Oct 31 13:34:26.831547 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.831513    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-openshift-service-ca\") pod \"maas-api-e2e-api-b-752869-6f4447749-rthlq\" (UID: \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:34:26.831750 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.831724    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-metrics-certs\") pod \"maas-api-e2e-api-b-752869-6f4447749-rthlq\" (UID: \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:34:26.831842 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.831822    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-maas-api-tls\") pod \"maas-api-e2e-api-b-752869-6f4447749-rthlq\" (UID: \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:34:26.831946 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.831929    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8sh\" (UniqueName: \"kubernetes.io/projected/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-kube-api-access-qx8sh\") pod \"maas-api-e2e-api-b-752869-6f4447749-rthlq\" (UID: \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:34:26.934463 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.934426    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-metrics-certs\") pod \"maas-api-e2e-api-b-752869-6f4447749-rthlq\" (UID: \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:34:26.934649 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.934514    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-maas-api-tls\") pod \"maas-api-e2e-api-b-752869-6f4447749-rthlq\" (UID: \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:34:26.934649 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.934628    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8sh\" (UniqueName: \"kubernetes.io/projected/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-kube-api-access-qx8sh\") pod \"maas-api-e2e-api-b-752869-6f4447749-rthlq\" (UID: \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:34:26.934829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.934814    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-openshift-service-ca\") pod \"maas-api-e2e-api-b-752869-6f4447749-rthlq\" (UID: \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:34:26.935777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.935748    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-openshift-service-ca\") pod \"maas-api-e2e-api-b-752869-6f4447749-rthlq\" (UID: \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:34:26.939927 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.939859    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-maas-api-tls\") pod \"maas-api-e2e-api-b-752869-6f4447749-rthlq\" (UID: \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:34:26.947984 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.947864    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-metrics-certs\") pod \"maas-api-e2e-api-b-752869-6f4447749-rthlq\" (UID: \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:34:26.950901 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:26.950871    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8sh\" (UniqueName: \"kubernetes.io/projected/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-kube-api-access-qx8sh\") pod \"maas-api-e2e-api-b-752869-6f4447749-rthlq\" (UID: \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:34:27.020382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:27.020339    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:34:27.494119 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:27.494083    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"]
Oct 31 13:34:27.870538 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:27.870502    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq" event={"ID":"f1fe6bcb-7c06-423e-88d6-0efe11f984b0","Type":"ContainerStarted","Data":"6e14057e8a9c1ac8cecde59ca36febc5f08e2ca490fa3ca53719d051b95b1c7a"}
Oct 31 13:34:28.880572 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:28.880531    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq" event={"ID":"f1fe6bcb-7c06-423e-88d6-0efe11f984b0","Type":"ContainerStarted","Data":"1784ba48f6ba5f22e7a1604f3605a8e307eb4089fe890ced19eb78366261169f"}
Oct 31 13:34:28.881089 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:28.880613    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:34:28.943057 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:28.942996    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq" podStartSLOduration=2.465928722 podStartE2EDuration="2.942976042s" totalImagesPullingTime="477.04732ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:28.928817847 +0000 UTC m=+1868.547477012" watchObservedRunningTime="2026-10-31 13:34:28.942976042 +0000 UTC m=+1868.561635207"
Oct 31 13:34:30.570848 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.570810    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"]
Oct 31 13:34:30.575782 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.575753    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"]
Oct 31 13:34:30.575934 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.575883    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.581064 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.581033    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cce3e7-openshift-default-dockercfg-x7qmg\""
Oct 31 13:34:30.696555 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.696518    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-credential-socket\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.722317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.696919    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d5ff03-7667-4572-9447-e2972398e04e-istio-podinfo\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.722768 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.722732    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-istio-data\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.722979 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.722960    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-istio-envoy\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.729325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.723269    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-workload-socket\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.729537 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.729509    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d5ff03-7667-4572-9447-e2972398e04e-istio-token\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.729621 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.729571    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-workload-certs\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.729703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.729686    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d5ff03-7667-4572-9447-e2972398e04e-istiod-ca-cert\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.729793 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.729773    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmxn5\" (UniqueName: \"kubernetes.io/projected/c5d5ff03-7667-4572-9447-e2972398e04e-kube-api-access-qmxn5\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.836373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.835360    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-istio-data\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.836373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.835466    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-istio-envoy\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.836373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.835646    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-workload-socket\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.836373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.835907    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d5ff03-7667-4572-9447-e2972398e04e-istio-token\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.836373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.835941    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-workload-certs\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.836373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.836015    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d5ff03-7667-4572-9447-e2972398e04e-istiod-ca-cert\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.836373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.836084    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmxn5\" (UniqueName: \"kubernetes.io/projected/c5d5ff03-7667-4572-9447-e2972398e04e-kube-api-access-qmxn5\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.852281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.850759    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-workload-certs\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.852281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.851853    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d5ff03-7667-4572-9447-e2972398e04e-istiod-ca-cert\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.852281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.852101    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-istio-data\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.853511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.853486    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-credential-socket\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.853784 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.853762    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d5ff03-7667-4572-9447-e2972398e04e-istio-podinfo\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.857656 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.855345    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-credential-socket\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.861402 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.855967    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-workload-socket\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.894566 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.890721    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-istio-envoy\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.917907 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.916580    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d5ff03-7667-4572-9447-e2972398e04e-istio-podinfo\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.921018 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.920943    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmxn5\" (UniqueName: \"kubernetes.io/projected/c5d5ff03-7667-4572-9447-e2972398e04e-kube-api-access-qmxn5\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:30.922073 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:30.921969    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d5ff03-7667-4572-9447-e2972398e04e-istio-token\") pod \"e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") " pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:31.193349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:31.193262    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:31.523682 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:31.523591    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"]
Oct 31 13:34:31.528730 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:34:31.528691    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d5ff03_7667_4572_9447_e2972398e04e.slice/crio-cad2714d521b6cae79ed9f6d47ba32ff9a85c8808af1b0daa78e5782e66fcf31 WatchSource:0}: Error finding container cad2714d521b6cae79ed9f6d47ba32ff9a85c8808af1b0daa78e5782e66fcf31: Status 404 returned error can't find the container with id cad2714d521b6cae79ed9f6d47ba32ff9a85c8808af1b0daa78e5782e66fcf31
Oct 31 13:34:31.532449 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:31.532408    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:34:31.532558 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:31.532532    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:34:31.532604 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:31.532578    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:34:31.946068 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:31.946018    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" event={"ID":"c5d5ff03-7667-4572-9447-e2972398e04e","Type":"ContainerStarted","Data":"3c3f876906cb56e137fa72e5d512b21b4e1bc9cedcadac824c6705e35f9ded66"}
Oct 31 13:34:31.946546 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:31.946095    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" event={"ID":"c5d5ff03-7667-4572-9447-e2972398e04e","Type":"ContainerStarted","Data":"cad2714d521b6cae79ed9f6d47ba32ff9a85c8808af1b0daa78e5782e66fcf31"}
Oct 31 13:34:31.974079 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:31.974017    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" podStartSLOduration=1.974000687 podStartE2EDuration="1.974000687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:31.970036297 +0000 UTC m=+1871.588695575" watchObservedRunningTime="2026-10-31 13:34:31.974000687 +0000 UTC m=+1871.592659850"
Oct 31 13:34:32.194371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:32.194335    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:32.200779 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:32.200707    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:32.954369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:32.954335    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:32.955357 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:32.955335    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:34:34.903098 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:34.903070    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:34:39.199313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:39.198441    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:39.345438 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:39.345377    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:39.406349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:39.406274    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:39.434486 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:39.434441    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:39.509731 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:39.509616    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:39.541073 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:39.541031    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:34:39.622135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:39.603354    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:39.622135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:39.603724    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:39.685483 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:39.684320    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:39.768830 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:39.756266    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:40.209324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:40.209067    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:40.464369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:40.464212    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:40.486864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:40.464727    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:34:40.497443 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:40.491999    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:40.590317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:40.580660    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:34:40.590317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:40.588439    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:40.655526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:40.655490    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:34:40.668535 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:40.668477    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:34:40.678542 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:40.678506    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:34:40.679326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:40.679276    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:40.737563 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:40.737463    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:40.769731 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:40.769683    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:34:40.834858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:40.834812    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq" podUID="c1a40e94-c560-4f09-b721-533261b4ff25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:40.907588 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:40.907559    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:34:41.137335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.137277    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:34:41.137637 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.137620    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:34:41.188728 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.161109    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:34:41.192440 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.192415    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:34:41.222428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.222397    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:34:41.222938 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.222466    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:34:41.222938 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.210607    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded"
Oct 31 13:34:41.237829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.211051    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5c26q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:34:41.237829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.223145    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:41.420050 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.419956    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:41.490322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.482595    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:34:41.518545 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.518489    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:34:41.668891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.668849    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:34:41.801412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.794736    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:41.862256 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.862224    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:34:41.919520 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.916012    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq" podUID="c1a40e94-c560-4f09-b721-533261b4ff25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:41.919520 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.916460    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:34:41.919520 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.916760    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:34:41.919520 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.917022    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:34:41.919520 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.917278    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:34:41.919520 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.918642    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded"
Oct 31 13:34:41.919520 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.918940    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:34:41.938316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.935915    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:34:41.938316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:41.936666    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:42.092647 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:42.092567    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:34:42.127344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:42.124039    4660 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 13:34:42.190318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:42.182032    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:34:42.215586 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:42.215550    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:34:42.247410 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:42.247252    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:34:42.290090 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:42.290061    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:34:42.393176 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:42.392866    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:34:42.403460 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:42.403434    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:34:42.492322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:42.489541    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:34:42.492322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:42.490936    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:34:47.333486 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:47.333438    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:34:47.334183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:47.333526    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn" podUID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:34:49.062199 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.061433    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"]
Oct 31 13:34:49.075281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.075182    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.084230 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.080681    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"]
Oct 31 13:34:49.092722 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.092693    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-580f24-openshift-default-dockercfg-98tzq\""
Oct 31 13:34:49.368507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.299205    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9smhk\" (UniqueName: \"kubernetes.io/projected/d86a5000-770e-409d-b23a-f836beb99b08-kube-api-access-9smhk\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.368507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.299267    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-workload-socket\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.368507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.299417    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d86a5000-770e-409d-b23a-f836beb99b08-istio-token\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.368507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.299490    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-istio-data\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.368507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.299529    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86a5000-770e-409d-b23a-f836beb99b08-istio-podinfo\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.368507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.299581    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86a5000-770e-409d-b23a-f836beb99b08-istiod-ca-cert\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.368507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.299626    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-workload-certs\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.368507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.299737    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-istio-envoy\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.368507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.299834    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-credential-socket\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.401231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.401188    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-credential-socket\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.401488 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.401341    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9smhk\" (UniqueName: \"kubernetes.io/projected/d86a5000-770e-409d-b23a-f836beb99b08-kube-api-access-9smhk\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.401488 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.401375    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-workload-socket\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.402915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.401483    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d86a5000-770e-409d-b23a-f836beb99b08-istio-token\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.402915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.401547    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-istio-data\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.402915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.401579    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86a5000-770e-409d-b23a-f836beb99b08-istio-podinfo\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.402915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.401637    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86a5000-770e-409d-b23a-f836beb99b08-istiod-ca-cert\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.402915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.401681    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-workload-certs\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.402915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.401781    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-credential-socket\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.402915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.401788    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-istio-envoy\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.402915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.402225    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-istio-data\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.404818 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.402804    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-workload-socket\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.404921 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.403873    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-workload-certs\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.404921 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.404595    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86a5000-770e-409d-b23a-f836beb99b08-istiod-ca-cert\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.404921 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.404774    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-istio-envoy\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.409286 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.407802    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86a5000-770e-409d-b23a-f836beb99b08-istio-podinfo\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.422288 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.422057    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d86a5000-770e-409d-b23a-f836beb99b08-istio-token\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.422288 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.422241    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9smhk\" (UniqueName: \"kubernetes.io/projected/d86a5000-770e-409d-b23a-f836beb99b08-kube-api-access-9smhk\") pod \"e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") " pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:49.718941 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:49.718896    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:50.029626 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:34:50.029595    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86a5000_770e_409d_b23a_f836beb99b08.slice/crio-72fb24090d9f616cabc08ef15cc765fee8140089419130eb694787619758a739 WatchSource:0}: Error finding container 72fb24090d9f616cabc08ef15cc765fee8140089419130eb694787619758a739: Status 404 returned error can't find the container with id 72fb24090d9f616cabc08ef15cc765fee8140089419130eb694787619758a739
Oct 31 13:34:50.033951 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:50.033461    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:34:50.033951 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:50.033553    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:34:50.033951 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:50.033606    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:34:50.033951 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:50.033720    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"]
Oct 31 13:34:50.383461 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:50.383420    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" event={"ID":"d86a5000-770e-409d-b23a-f836beb99b08","Type":"ContainerStarted","Data":"26aaf65ec927aef941594f84586d17582de30dd442b3920221eb17f2babc4506"}
Oct 31 13:34:50.383938 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:50.383493    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" event={"ID":"d86a5000-770e-409d-b23a-f836beb99b08","Type":"ContainerStarted","Data":"72fb24090d9f616cabc08ef15cc765fee8140089419130eb694787619758a739"}
Oct 31 13:34:50.386329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:50.386268    4660 generic.go:356] "Generic (PLEG): container finished" podID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerID="5f72848048b7bb19b1b96647657945c5c2563c2767c3bb671e6f5d90a79e7d72" exitCode=0
Oct 31 13:34:50.386329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:50.386315    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn" event={"ID":"e71b679c-5dff-4c19-bef4-a4bf19093b34","Type":"ContainerDied","Data":"5f72848048b7bb19b1b96647657945c5c2563c2767c3bb671e6f5d90a79e7d72"}
Oct 31 13:34:50.386724 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:50.386706    4660 scope.go:122] "RemoveContainer" containerID="5f72848048b7bb19b1b96647657945c5c2563c2767c3bb671e6f5d90a79e7d72"
Oct 31 13:34:50.388882 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:50.388853    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:34:50.388958 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:50.388943    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:34:50.389028 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:50.389013    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:34:50.450928 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:50.450188    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" podStartSLOduration=1.4501690329999999 podStartE2EDuration="1.450169033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:50.418496298 +0000 UTC m=+1890.037155461" watchObservedRunningTime="2026-10-31 13:34:50.450169033 +0000 UTC m=+1890.068828198"
Oct 31 13:34:50.723276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:50.723193    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:50.840017 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:34:50.839971    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:34:51.395415 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:51.395380    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn" event={"ID":"e71b679c-5dff-4c19-bef4-a4bf19093b34","Type":"ContainerStarted","Data":"54424b617f2a42315c52130074f9dbdf199804907190e469deea62624f056a22"}
Oct 31 13:34:51.784834 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:51.784747    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:52.120728 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:52.120691    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:34:52.121066 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:52.121041    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 13:34:52.121218 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:52.121196    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn" podUID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 13:34:52.438929 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:52.438841    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:52.439575 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:52.439553    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:34:53.125327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:53.124692    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 13:34:53.125327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:53.124757    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn" podUID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 13:34:54.131317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:54.128881    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 13:34:54.131317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:54.128945    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn" podUID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 13:34:55.120346 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:55.119714    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 13:34:55.120346 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:55.119779    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn" podUID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 13:34:56.117719 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:56.117672    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 13:34:56.118346 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:56.118282    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn" podUID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 13:34:57.120733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:57.120695    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 13:34:57.121394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:57.121360    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn" podUID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 13:34:58.124069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:58.124032    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:34:58.124593 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:58.124569    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:34:58.125456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:34:58.125427    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 13:35:05.053069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:05.053030    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11116-230099"]
Oct 31 13:35:05.060548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:05.060515    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-230099"
Oct 31 13:35:05.065542 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:05.065152    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11116-230099"]
Oct 31 13:35:05.065699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:05.065549    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-whw9x\""
Oct 31 13:35:05.193658 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:05.193622    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pmjb\" (UniqueName: \"kubernetes.io/projected/8a3fee1d-61a4-4081-9faa-89158a9a0e20-kube-api-access-5pmjb\") pod \"test-curl-11116-230099\" (UID: \"8a3fee1d-61a4-4081-9faa-89158a9a0e20\") " pod="opendatahub/test-curl-11116-230099"
Oct 31 13:35:05.295662 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:05.295620    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pmjb\" (UniqueName: \"kubernetes.io/projected/8a3fee1d-61a4-4081-9faa-89158a9a0e20-kube-api-access-5pmjb\") pod \"test-curl-11116-230099\" (UID: \"8a3fee1d-61a4-4081-9faa-89158a9a0e20\") " pod="opendatahub/test-curl-11116-230099"
Oct 31 13:35:05.308920 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:05.308848    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pmjb\" (UniqueName: \"kubernetes.io/projected/8a3fee1d-61a4-4081-9faa-89158a9a0e20-kube-api-access-5pmjb\") pod \"test-curl-11116-230099\" (UID: \"8a3fee1d-61a4-4081-9faa-89158a9a0e20\") " pod="opendatahub/test-curl-11116-230099"
Oct 31 13:35:05.376653 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:05.376623    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-230099"
Oct 31 13:35:05.663887 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:35:05.663861    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a3fee1d_61a4_4081_9faa_89158a9a0e20.slice/crio-d29624b5c0f7a65bb3dd4cfd95b7bb31cbc0ab44c7e8de199d4ed003bb06df6e WatchSource:0}: Error finding container d29624b5c0f7a65bb3dd4cfd95b7bb31cbc0ab44c7e8de199d4ed003bb06df6e: Status 404 returned error can't find the container with id d29624b5c0f7a65bb3dd4cfd95b7bb31cbc0ab44c7e8de199d4ed003bb06df6e
Oct 31 13:35:05.665569 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:05.665412    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11116-230099"]
Oct 31 13:35:06.633883 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:06.633556    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11116-230099" event={"ID":"8a3fee1d-61a4-4081-9faa-89158a9a0e20","Type":"ContainerStarted","Data":"7c03798a3c3d6e3a06e4fae438698fd00349a82d94be65ed4e8e1092b3c71744"}
Oct 31 13:35:06.634379 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:06.633909    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11116-230099" event={"ID":"8a3fee1d-61a4-4081-9faa-89158a9a0e20","Type":"ContainerStarted","Data":"d29624b5c0f7a65bb3dd4cfd95b7bb31cbc0ab44c7e8de199d4ed003bb06df6e"}
Oct 31 13:35:06.652889 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:06.652828    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11116-230099" podStartSLOduration=1.6528111970000001 podStartE2EDuration="1.652811197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:06.649665531 +0000 UTC m=+1906.268324696" watchObservedRunningTime="2026-10-31 13:35:06.652811197 +0000 UTC m=+1906.271470361"
Oct 31 13:35:07.353010 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:07.352975    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11116-230099"]
Oct 31 13:35:07.360521 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:07.360492    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11116-230099"]
Oct 31 13:35:07.549316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:07.548646    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11116-0448cc"]
Oct 31 13:35:07.552332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:07.550112    4660 cpu_manager.go:414] "containerMap: removing container" podUID="8a3fee1d-61a4-4081-9faa-89158a9a0e20" containerName="test-curl-11116-230099"
Oct 31 13:35:07.552332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:07.550139    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3fee1d-61a4-4081-9faa-89158a9a0e20" containerName="test-curl-11116-230099"
Oct 31 13:35:07.552332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:07.550569    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3fee1d-61a4-4081-9faa-89158a9a0e20" containerName="test-curl-11116-230099"
Oct 31 13:35:07.562067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:07.562040    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11116-0448cc"]
Oct 31 13:35:07.562207 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:07.562129    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-0448cc"
Oct 31 13:35:07.748885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:07.748838    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8hjs\" (UniqueName: \"kubernetes.io/projected/4d450a10-b87b-42ae-b496-a8f8249cebb1-kube-api-access-z8hjs\") pod \"test-curl-11116-0448cc\" (UID: \"4d450a10-b87b-42ae-b496-a8f8249cebb1\") " pod="opendatahub/test-curl-11116-0448cc"
Oct 31 13:35:07.850613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:07.850581    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8hjs\" (UniqueName: \"kubernetes.io/projected/4d450a10-b87b-42ae-b496-a8f8249cebb1-kube-api-access-z8hjs\") pod \"test-curl-11116-0448cc\" (UID: \"4d450a10-b87b-42ae-b496-a8f8249cebb1\") " pod="opendatahub/test-curl-11116-0448cc"
Oct 31 13:35:07.863848 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:07.863809    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8hjs\" (UniqueName: \"kubernetes.io/projected/4d450a10-b87b-42ae-b496-a8f8249cebb1-kube-api-access-z8hjs\") pod \"test-curl-11116-0448cc\" (UID: \"4d450a10-b87b-42ae-b496-a8f8249cebb1\") " pod="opendatahub/test-curl-11116-0448cc"
Oct 31 13:35:07.878197 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:07.877508    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-0448cc"
Oct 31 13:35:08.242443 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:08.242410    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11116-0448cc"]
Oct 31 13:35:08.669410 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:08.667030    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11116-230099" podUID="8a3fee1d-61a4-4081-9faa-89158a9a0e20" containerName="test-curl-11116-230099" containerID="cri-o://7c03798a3c3d6e3a06e4fae438698fd00349a82d94be65ed4e8e1092b3c71744" gracePeriod=2
Oct 31 13:35:08.669410 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:08.668393    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11116-0448cc" event={"ID":"4d450a10-b87b-42ae-b496-a8f8249cebb1","Type":"ContainerStarted","Data":"c8e1f1e087db64c92b2643471fd029aec2e03e61e502b6318608b5fbd30fd712"}
Oct 31 13:35:08.669410 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:08.668435    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11116-0448cc" event={"ID":"4d450a10-b87b-42ae-b496-a8f8249cebb1","Type":"ContainerStarted","Data":"16b17b3ba3d2c86e036447bbb070c0d50e4149c43da0547bcdce3555473828bb"}
Oct 31 13:35:08.688665 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:08.688612    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11116-0448cc" podStartSLOduration=1.688597672 podStartE2EDuration="1.688597672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:08.687612786 +0000 UTC m=+1908.306271949" watchObservedRunningTime="2026-10-31 13:35:08.688597672 +0000 UTC m=+1908.307256836"
Oct 31 13:35:09.519857 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:09.519823    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11116-0448cc"]
Oct 31 13:35:09.527246 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:09.526848    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11116-0448cc"]
Oct 31 13:35:09.682260 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:09.682229    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11116-b2cfc4"]
Oct 31 13:35:09.683617 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:09.683585    4660 cpu_manager.go:414] "containerMap: removing container" podUID="4d450a10-b87b-42ae-b496-a8f8249cebb1" containerName="test-curl-11116-0448cc"
Oct 31 13:35:09.683617 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:09.683612    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d450a10-b87b-42ae-b496-a8f8249cebb1" containerName="test-curl-11116-0448cc"
Oct 31 13:35:09.684057 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:09.684034    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d450a10-b87b-42ae-b496-a8f8249cebb1" containerName="test-curl-11116-0448cc"
Oct 31 13:35:09.688214 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:09.688193    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-b2cfc4"
Oct 31 13:35:09.691156 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:09.689834    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11116-b2cfc4"]
Oct 31 13:35:09.765100 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:09.765056    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgh9g\" (UniqueName: \"kubernetes.io/projected/a45ddaae-f3be-42e1-a62e-1047614e9a0e-kube-api-access-hgh9g\") pod \"test-curl-11116-b2cfc4\" (UID: \"a45ddaae-f3be-42e1-a62e-1047614e9a0e\") " pod="opendatahub/test-curl-11116-b2cfc4"
Oct 31 13:35:09.866631 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:09.866553    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgh9g\" (UniqueName: \"kubernetes.io/projected/a45ddaae-f3be-42e1-a62e-1047614e9a0e-kube-api-access-hgh9g\") pod \"test-curl-11116-b2cfc4\" (UID: \"a45ddaae-f3be-42e1-a62e-1047614e9a0e\") " pod="opendatahub/test-curl-11116-b2cfc4"
Oct 31 13:35:09.880277 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:09.880253    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgh9g\" (UniqueName: \"kubernetes.io/projected/a45ddaae-f3be-42e1-a62e-1047614e9a0e-kube-api-access-hgh9g\") pod \"test-curl-11116-b2cfc4\" (UID: \"a45ddaae-f3be-42e1-a62e-1047614e9a0e\") " pod="opendatahub/test-curl-11116-b2cfc4"
Oct 31 13:35:10.014510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:10.014480    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-b2cfc4"
Oct 31 13:35:10.321038 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:10.321010    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11116-b2cfc4"]
Oct 31 13:35:10.321769 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:35:10.321747    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda45ddaae_f3be_42e1_a62e_1047614e9a0e.slice/crio-ed05e1334358f135d798b69ff909cd131efcd3c74bbe21dc0bdf96d9d332116b WatchSource:0}: Error finding container ed05e1334358f135d798b69ff909cd131efcd3c74bbe21dc0bdf96d9d332116b: Status 404 returned error can't find the container with id ed05e1334358f135d798b69ff909cd131efcd3c74bbe21dc0bdf96d9d332116b
Oct 31 13:35:10.697081 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:10.697036    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11116-0448cc" podUID="4d450a10-b87b-42ae-b496-a8f8249cebb1" containerName="test-curl-11116-0448cc" containerID="cri-o://c8e1f1e087db64c92b2643471fd029aec2e03e61e502b6318608b5fbd30fd712" gracePeriod=2
Oct 31 13:35:10.699352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:10.699321    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11116-b2cfc4" event={"ID":"a45ddaae-f3be-42e1-a62e-1047614e9a0e","Type":"ContainerStarted","Data":"1f730562b1f0bc252b725db06262b94a94921f6ccb5432b1fc22ea04e20f81e0"}
Oct 31 13:35:10.699515 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:10.699499    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11116-b2cfc4" event={"ID":"a45ddaae-f3be-42e1-a62e-1047614e9a0e","Type":"ContainerStarted","Data":"ed05e1334358f135d798b69ff909cd131efcd3c74bbe21dc0bdf96d9d332116b"}
Oct 31 13:35:10.756211 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:10.752988    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11116-b2cfc4" podStartSLOduration=1.752968263 podStartE2EDuration="1.752968263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:10.735279145 +0000 UTC m=+1910.353938322" watchObservedRunningTime="2026-10-31 13:35:10.752968263 +0000 UTC m=+1910.371627428"
Oct 31 13:35:11.195241 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.195170    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-230099"
Oct 31 13:35:11.311686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.311653    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3fee1d-61a4-4081-9faa-89158a9a0e20-kube-api-access-5pmjb\" (UniqueName: \"kubernetes.io/projected/8a3fee1d-61a4-4081-9faa-89158a9a0e20-kube-api-access-5pmjb\") pod \"8a3fee1d-61a4-4081-9faa-89158a9a0e20\" (UID: \"8a3fee1d-61a4-4081-9faa-89158a9a0e20\") "
Oct 31 13:35:11.314410 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.314371    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3fee1d-61a4-4081-9faa-89158a9a0e20-kube-api-access-5pmjb" pod "8a3fee1d-61a4-4081-9faa-89158a9a0e20" (UID: "8a3fee1d-61a4-4081-9faa-89158a9a0e20"). InnerVolumeSpecName "kube-api-access-5pmjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:35:11.414380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.414344    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pmjb\" (UniqueName: \"kubernetes.io/projected/8a3fee1d-61a4-4081-9faa-89158a9a0e20-kube-api-access-5pmjb\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:11.455070 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.454984    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11116-b2cfc4"]
Oct 31 13:35:11.460894 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.460865    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11116-b2cfc4"]
Oct 31 13:35:11.609970 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.609938    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11116-9a9316"]
Oct 31 13:35:11.611030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.611007    4660 cpu_manager.go:414] "containerMap: removing container" podUID="a45ddaae-f3be-42e1-a62e-1047614e9a0e" containerName="test-curl-11116-b2cfc4"
Oct 31 13:35:11.611088 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.611034    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a45ddaae-f3be-42e1-a62e-1047614e9a0e" containerName="test-curl-11116-b2cfc4"
Oct 31 13:35:11.611220 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.611206    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="a45ddaae-f3be-42e1-a62e-1047614e9a0e" containerName="test-curl-11116-b2cfc4"
Oct 31 13:35:11.618109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.618082    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-9a9316"
Oct 31 13:35:11.620000 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.619973    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11116-9a9316"]
Oct 31 13:35:11.708940 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.708851    4660 generic.go:356] "Generic (PLEG): container finished" podID="8a3fee1d-61a4-4081-9faa-89158a9a0e20" containerID="7c03798a3c3d6e3a06e4fae438698fd00349a82d94be65ed4e8e1092b3c71744" exitCode=137
Oct 31 13:35:11.708940 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.708914    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-230099"
Oct 31 13:35:11.709457 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.709018    4660 scope.go:122] "RemoveContainer" containerID="7c03798a3c3d6e3a06e4fae438698fd00349a82d94be65ed4e8e1092b3c71744"
Oct 31 13:35:11.719566 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.719537    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xppr4\" (UniqueName: \"kubernetes.io/projected/a9922ad2-b37d-45fb-8a49-1a209b2b6610-kube-api-access-xppr4\") pod \"test-curl-11116-9a9316\" (UID: \"a9922ad2-b37d-45fb-8a49-1a209b2b6610\") " pod="opendatahub/test-curl-11116-9a9316"
Oct 31 13:35:11.721471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.721449    4660 scope.go:122] "RemoveContainer" containerID="7c03798a3c3d6e3a06e4fae438698fd00349a82d94be65ed4e8e1092b3c71744"
Oct 31 13:35:11.721917 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:35:11.721887    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c03798a3c3d6e3a06e4fae438698fd00349a82d94be65ed4e8e1092b3c71744\": container with ID starting with 7c03798a3c3d6e3a06e4fae438698fd00349a82d94be65ed4e8e1092b3c71744 not found: ID does not exist" containerID="7c03798a3c3d6e3a06e4fae438698fd00349a82d94be65ed4e8e1092b3c71744"
Oct 31 13:35:11.722002 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.721923    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c03798a3c3d6e3a06e4fae438698fd00349a82d94be65ed4e8e1092b3c71744"} err="failed to get container status \"7c03798a3c3d6e3a06e4fae438698fd00349a82d94be65ed4e8e1092b3c71744\": rpc error: code = NotFound desc = could not find container \"7c03798a3c3d6e3a06e4fae438698fd00349a82d94be65ed4e8e1092b3c71744\": container with ID starting with 7c03798a3c3d6e3a06e4fae438698fd00349a82d94be65ed4e8e1092b3c71744 not found: ID does not exist"
Oct 31 13:35:11.724850 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:35:11.724814    4660 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11116-230099\" is forbidden: User \"system:node:ip-10-0-137-77.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-77.us-east-2.compute.internal' and this object" podUID="8a3fee1d-61a4-4081-9faa-89158a9a0e20" pod="opendatahub/test-curl-11116-230099"
Oct 31 13:35:11.821745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.821702    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xppr4\" (UniqueName: \"kubernetes.io/projected/a9922ad2-b37d-45fb-8a49-1a209b2b6610-kube-api-access-xppr4\") pod \"test-curl-11116-9a9316\" (UID: \"a9922ad2-b37d-45fb-8a49-1a209b2b6610\") " pod="opendatahub/test-curl-11116-9a9316"
Oct 31 13:35:11.834794 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.834768    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xppr4\" (UniqueName: \"kubernetes.io/projected/a9922ad2-b37d-45fb-8a49-1a209b2b6610-kube-api-access-xppr4\") pod \"test-curl-11116-9a9316\" (UID: \"a9922ad2-b37d-45fb-8a49-1a209b2b6610\") " pod="opendatahub/test-curl-11116-9a9316"
Oct 31 13:35:11.939059 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:11.939027    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-9a9316"
Oct 31 13:35:12.267046 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:35:12.267020    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9922ad2_b37d_45fb_8a49_1a209b2b6610.slice/crio-a3f8e391cf4b00237d305e82f85b248b5af39d4e123d1389afe827eb2bc8ef64 WatchSource:0}: Error finding container a3f8e391cf4b00237d305e82f85b248b5af39d4e123d1389afe827eb2bc8ef64: Status 404 returned error can't find the container with id a3f8e391cf4b00237d305e82f85b248b5af39d4e123d1389afe827eb2bc8ef64
Oct 31 13:35:12.267554 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:12.267528    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11116-9a9316"]
Oct 31 13:35:12.534502 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:12.534406    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3fee1d-61a4-4081-9faa-89158a9a0e20" path="/var/lib/kubelet/pods/8a3fee1d-61a4-4081-9faa-89158a9a0e20/volumes"
Oct 31 13:35:12.724231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:12.724187    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11116-b2cfc4" podUID="a45ddaae-f3be-42e1-a62e-1047614e9a0e" containerName="test-curl-11116-b2cfc4" containerID="cri-o://1f730562b1f0bc252b725db06262b94a94921f6ccb5432b1fc22ea04e20f81e0" gracePeriod=2
Oct 31 13:35:12.725388 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:12.725349    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11116-9a9316" event={"ID":"a9922ad2-b37d-45fb-8a49-1a209b2b6610","Type":"ContainerStarted","Data":"943aacb05bf2c4dbcca68b3f1a437a7a6098d0dfd38c1851aae646ce48493d6f"}
Oct 31 13:35:12.725469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:12.725396    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11116-9a9316" event={"ID":"a9922ad2-b37d-45fb-8a49-1a209b2b6610","Type":"ContainerStarted","Data":"a3f8e391cf4b00237d305e82f85b248b5af39d4e123d1389afe827eb2bc8ef64"}
Oct 31 13:35:12.745789 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:12.745653    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11116-9a9316" podStartSLOduration=1.745640672 podStartE2EDuration="1.745640672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:12.742159083 +0000 UTC m=+1912.360818247" watchObservedRunningTime="2026-10-31 13:35:12.745640672 +0000 UTC m=+1912.364299835"
Oct 31 13:35:13.028270 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:13.028243    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-0448cc"
Oct 31 13:35:13.157835 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:13.157804    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d450a10-b87b-42ae-b496-a8f8249cebb1-kube-api-access-z8hjs\" (UniqueName: \"kubernetes.io/projected/4d450a10-b87b-42ae-b496-a8f8249cebb1-kube-api-access-z8hjs\") pod \"4d450a10-b87b-42ae-b496-a8f8249cebb1\" (UID: \"4d450a10-b87b-42ae-b496-a8f8249cebb1\") "
Oct 31 13:35:13.159923 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:13.159895    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d450a10-b87b-42ae-b496-a8f8249cebb1-kube-api-access-z8hjs" pod "4d450a10-b87b-42ae-b496-a8f8249cebb1" (UID: "4d450a10-b87b-42ae-b496-a8f8249cebb1"). InnerVolumeSpecName "kube-api-access-z8hjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:35:13.259954 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:13.259920    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8hjs\" (UniqueName: \"kubernetes.io/projected/4d450a10-b87b-42ae-b496-a8f8249cebb1-kube-api-access-z8hjs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:13.448140 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:13.448056    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11116-9a9316"]
Oct 31 13:35:13.456103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:13.456074    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11116-9a9316"]
Oct 31 13:35:13.737086 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:13.736995    4660 generic.go:356] "Generic (PLEG): container finished" podID="4d450a10-b87b-42ae-b496-a8f8249cebb1" containerID="c8e1f1e087db64c92b2643471fd029aec2e03e61e502b6318608b5fbd30fd712" exitCode=137
Oct 31 13:35:13.737086 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:13.737032    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-0448cc"
Oct 31 13:35:13.737603 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:13.737087    4660 scope.go:122] "RemoveContainer" containerID="c8e1f1e087db64c92b2643471fd029aec2e03e61e502b6318608b5fbd30fd712"
Oct 31 13:35:13.750137 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:35:13.750110    4660 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11116-0448cc\" is forbidden: User \"system:node:ip-10-0-137-77.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-77.us-east-2.compute.internal' and this object" podUID="4d450a10-b87b-42ae-b496-a8f8249cebb1" pod="opendatahub/test-curl-11116-0448cc"
Oct 31 13:35:13.750733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:13.750709    4660 scope.go:122] "RemoveContainer" containerID="c8e1f1e087db64c92b2643471fd029aec2e03e61e502b6318608b5fbd30fd712"
Oct 31 13:35:13.751018 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:35:13.751000    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e1f1e087db64c92b2643471fd029aec2e03e61e502b6318608b5fbd30fd712\": container with ID starting with c8e1f1e087db64c92b2643471fd029aec2e03e61e502b6318608b5fbd30fd712 not found: ID does not exist" containerID="c8e1f1e087db64c92b2643471fd029aec2e03e61e502b6318608b5fbd30fd712"
Oct 31 13:35:13.751079 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:13.751033    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e1f1e087db64c92b2643471fd029aec2e03e61e502b6318608b5fbd30fd712"} err="failed to get container status \"c8e1f1e087db64c92b2643471fd029aec2e03e61e502b6318608b5fbd30fd712\": rpc error: code = NotFound desc = could not find container \"c8e1f1e087db64c92b2643471fd029aec2e03e61e502b6318608b5fbd30fd712\": container with ID starting with c8e1f1e087db64c92b2643471fd029aec2e03e61e502b6318608b5fbd30fd712 not found: ID does not exist"
Oct 31 13:35:14.529445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.529406    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d450a10-b87b-42ae-b496-a8f8249cebb1" path="/var/lib/kubelet/pods/4d450a10-b87b-42ae-b496-a8f8249cebb1/volumes"
Oct 31 13:35:14.552045 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.552010    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"]
Oct 31 13:35:14.553103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.553076    4660 cpu_manager.go:414] "containerMap: removing container" podUID="a9922ad2-b37d-45fb-8a49-1a209b2b6610" containerName="test-curl-11116-9a9316"
Oct 31 13:35:14.553103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.553104    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9922ad2-b37d-45fb-8a49-1a209b2b6610" containerName="test-curl-11116-9a9316"
Oct 31 13:35:14.553381 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.553363    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9922ad2-b37d-45fb-8a49-1a209b2b6610" containerName="test-curl-11116-9a9316"
Oct 31 13:35:14.559066 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.559043    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.565774 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.565742    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6aa924-openshift-default-dockercfg-fgb76\""
Oct 31 13:35:14.582571 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.581511    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"]
Oct 31 13:35:14.687674 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.686462    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6249baed-5e62-4178-a9a8-ffe554407c0b-istiod-ca-cert\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.687674 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.686538    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-workload-certs\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.687674 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.686614    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-envoy\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.687674 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.686658    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-token\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.687674 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.686789    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsd4x\" (UniqueName: \"kubernetes.io/projected/6249baed-5e62-4178-a9a8-ffe554407c0b-kube-api-access-zsd4x\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.687674 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.686840    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-workload-socket\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.687674 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.686906    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-data\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.687674 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.686973    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-podinfo\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.687674 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.687026    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-credential-socket\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.748589 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.748553    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11116-9a9316" podUID="a9922ad2-b37d-45fb-8a49-1a209b2b6610" containerName="test-curl-11116-9a9316" containerID="cri-o://943aacb05bf2c4dbcca68b3f1a437a7a6098d0dfd38c1851aae646ce48493d6f" gracePeriod=2
Oct 31 13:35:14.795412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.793749    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6249baed-5e62-4178-a9a8-ffe554407c0b-istiod-ca-cert\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.795412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.793828    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-workload-certs\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.795412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.793905    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-envoy\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.795412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.793949    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-token\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.795412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.794081    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsd4x\" (UniqueName: \"kubernetes.io/projected/6249baed-5e62-4178-a9a8-ffe554407c0b-kube-api-access-zsd4x\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.795412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.794133    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-workload-socket\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.795412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.794197    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-data\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.795412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.794268    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-podinfo\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.795412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.794345    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-credential-socket\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.795412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.795068    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-credential-socket\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.808850 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.796068    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6249baed-5e62-4178-a9a8-ffe554407c0b-istiod-ca-cert\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.808850 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.796344    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-workload-certs\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.808850 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.796925    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-workload-socket\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.808850 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.797680    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-data\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.841913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.841844    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-token\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.863328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.853839    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-podinfo\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.863328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.854207    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-envoy\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.889935 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.888534    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsd4x\" (UniqueName: \"kubernetes.io/projected/6249baed-5e62-4178-a9a8-ffe554407c0b-kube-api-access-zsd4x\") pod \"e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") " pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:14.889935 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:14.888839    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:15.472647 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:15.472456    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-b2cfc4"
Oct 31 13:35:15.496593 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:15.495493    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a45ddaae-f3be-42e1-a62e-1047614e9a0e-kube-api-access-hgh9g\" (UniqueName: \"kubernetes.io/projected/a45ddaae-f3be-42e1-a62e-1047614e9a0e-kube-api-access-hgh9g\") pod \"a45ddaae-f3be-42e1-a62e-1047614e9a0e\" (UID: \"a45ddaae-f3be-42e1-a62e-1047614e9a0e\") "
Oct 31 13:35:15.502998 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:15.502969    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45ddaae-f3be-42e1-a62e-1047614e9a0e-kube-api-access-hgh9g" pod "a45ddaae-f3be-42e1-a62e-1047614e9a0e" (UID: "a45ddaae-f3be-42e1-a62e-1047614e9a0e"). InnerVolumeSpecName "kube-api-access-hgh9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:35:15.601432 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:15.599554    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgh9g\" (UniqueName: \"kubernetes.io/projected/a45ddaae-f3be-42e1-a62e-1047614e9a0e-kube-api-access-hgh9g\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:15.607599 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:15.607562    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"]
Oct 31 13:35:15.610407 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:15.610359    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:35:15.610504 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:15.610452    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:35:15.610561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:15.610498    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:35:15.807010 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:15.806573    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" event={"ID":"6249baed-5e62-4178-a9a8-ffe554407c0b","Type":"ContainerStarted","Data":"a6e2db0b5fa2e9b8f58a788e2535b7d1579df87de8422c7fb67c758016a9e3af"}
Oct 31 13:35:15.809492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:15.808814    4660 generic.go:356] "Generic (PLEG): container finished" podID="a45ddaae-f3be-42e1-a62e-1047614e9a0e" containerID="1f730562b1f0bc252b725db06262b94a94921f6ccb5432b1fc22ea04e20f81e0" exitCode=137
Oct 31 13:35:15.809492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:15.808879    4660 scope.go:122] "RemoveContainer" containerID="1f730562b1f0bc252b725db06262b94a94921f6ccb5432b1fc22ea04e20f81e0"
Oct 31 13:35:15.809492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:15.809046    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-b2cfc4"
Oct 31 13:35:15.832153 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:35:15.832116    4660 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11116-b2cfc4\" is forbidden: User \"system:node:ip-10-0-137-77.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-77.us-east-2.compute.internal' and this object" podUID="a45ddaae-f3be-42e1-a62e-1047614e9a0e" pod="opendatahub/test-curl-11116-b2cfc4"
Oct 31 13:35:15.841504 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:15.841478    4660 scope.go:122] "RemoveContainer" containerID="1f730562b1f0bc252b725db06262b94a94921f6ccb5432b1fc22ea04e20f81e0"
Oct 31 13:35:15.841861 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:35:15.841836    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f730562b1f0bc252b725db06262b94a94921f6ccb5432b1fc22ea04e20f81e0\": container with ID starting with 1f730562b1f0bc252b725db06262b94a94921f6ccb5432b1fc22ea04e20f81e0 not found: ID does not exist" containerID="1f730562b1f0bc252b725db06262b94a94921f6ccb5432b1fc22ea04e20f81e0"
Oct 31 13:35:15.841946 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:15.841872    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f730562b1f0bc252b725db06262b94a94921f6ccb5432b1fc22ea04e20f81e0"} err="failed to get container status \"1f730562b1f0bc252b725db06262b94a94921f6ccb5432b1fc22ea04e20f81e0\": rpc error: code = NotFound desc = could not find container \"1f730562b1f0bc252b725db06262b94a94921f6ccb5432b1fc22ea04e20f81e0\": container with ID starting with 1f730562b1f0bc252b725db06262b94a94921f6ccb5432b1fc22ea04e20f81e0 not found: ID does not exist"
Oct 31 13:35:16.659801 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:16.659764    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a45ddaae-f3be-42e1-a62e-1047614e9a0e" path="/var/lib/kubelet/pods/a45ddaae-f3be-42e1-a62e-1047614e9a0e/volumes"
Oct 31 13:35:16.826091 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:16.826049    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" event={"ID":"6249baed-5e62-4178-a9a8-ffe554407c0b","Type":"ContainerStarted","Data":"9d9a900557fc97ec5245a5cd81af8edb12e443771052545a7076b9191d5cffd8"}
Oct 31 13:35:16.861014 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:16.860942    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" podStartSLOduration=2.860922293 podStartE2EDuration="2.860922293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:16.85149509 +0000 UTC m=+1916.470154256" watchObservedRunningTime="2026-10-31 13:35:16.860922293 +0000 UTC m=+1916.479581459"
Oct 31 13:35:16.889728 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:16.889688    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:16.896551 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:16.896526    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:17.147938 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:17.147824    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-9a9316"
Oct 31 13:35:17.182389 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:17.181689    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9922ad2-b37d-45fb-8a49-1a209b2b6610-kube-api-access-xppr4\" (UniqueName: \"kubernetes.io/projected/a9922ad2-b37d-45fb-8a49-1a209b2b6610-kube-api-access-xppr4\") pod \"a9922ad2-b37d-45fb-8a49-1a209b2b6610\" (UID: \"a9922ad2-b37d-45fb-8a49-1a209b2b6610\") "
Oct 31 13:35:17.186579 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:17.186005    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9922ad2-b37d-45fb-8a49-1a209b2b6610-kube-api-access-xppr4" pod "a9922ad2-b37d-45fb-8a49-1a209b2b6610" (UID: "a9922ad2-b37d-45fb-8a49-1a209b2b6610"). InnerVolumeSpecName "kube-api-access-xppr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:35:17.286178 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:17.285576    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xppr4\" (UniqueName: \"kubernetes.io/projected/a9922ad2-b37d-45fb-8a49-1a209b2b6610-kube-api-access-xppr4\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:17.840915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:17.840875    4660 generic.go:356] "Generic (PLEG): container finished" podID="a9922ad2-b37d-45fb-8a49-1a209b2b6610" containerID="943aacb05bf2c4dbcca68b3f1a437a7a6098d0dfd38c1851aae646ce48493d6f" exitCode=137
Oct 31 13:35:17.841386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:17.840927    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11116-9a9316"
Oct 31 13:35:17.841386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:17.840965    4660 scope.go:122] "RemoveContainer" containerID="943aacb05bf2c4dbcca68b3f1a437a7a6098d0dfd38c1851aae646ce48493d6f"
Oct 31 13:35:17.841590 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:17.841573    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:17.846572 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:17.846549    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:35:17.859756 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:17.859732    4660 scope.go:122] "RemoveContainer" containerID="943aacb05bf2c4dbcca68b3f1a437a7a6098d0dfd38c1851aae646ce48493d6f"
Oct 31 13:35:17.868420 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:35:17.867711    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943aacb05bf2c4dbcca68b3f1a437a7a6098d0dfd38c1851aae646ce48493d6f\": container with ID starting with 943aacb05bf2c4dbcca68b3f1a437a7a6098d0dfd38c1851aae646ce48493d6f not found: ID does not exist" containerID="943aacb05bf2c4dbcca68b3f1a437a7a6098d0dfd38c1851aae646ce48493d6f"
Oct 31 13:35:17.868420 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:17.867752    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943aacb05bf2c4dbcca68b3f1a437a7a6098d0dfd38c1851aae646ce48493d6f"} err="failed to get container status \"943aacb05bf2c4dbcca68b3f1a437a7a6098d0dfd38c1851aae646ce48493d6f\": rpc error: code = NotFound desc = could not find container \"943aacb05bf2c4dbcca68b3f1a437a7a6098d0dfd38c1851aae646ce48493d6f\": container with ID starting with 943aacb05bf2c4dbcca68b3f1a437a7a6098d0dfd38c1851aae646ce48493d6f not found: ID does not exist"
Oct 31 13:35:17.879764 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:35:17.879734    4660 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11116-9a9316\" is forbidden: User \"system:node:ip-10-0-137-77.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-77.us-east-2.compute.internal' and this object" podUID="a9922ad2-b37d-45fb-8a49-1a209b2b6610" pod="opendatahub/test-curl-11116-9a9316"
Oct 31 13:35:18.531727 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:18.531681    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9922ad2-b37d-45fb-8a49-1a209b2b6610" path="/var/lib/kubelet/pods/a9922ad2-b37d-45fb-8a49-1a209b2b6610/volumes"
Oct 31 13:35:20.793014 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:35:20.792975    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:35:25.818319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:25.800419    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:25.867755 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:25.867683    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:25.938329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:25.932375    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:25.938329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:25.933332    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:25.938329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:25.933591    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:35:26.199319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:26.198471    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5c26q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:35:26.199319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:26.198569    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:26.231319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:26.230904    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:35:26.269418 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:26.261756    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq" podUID="c1a40e94-c560-4f09-b721-533261b4ff25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded"
Oct 31 13:35:26.300957 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:26.300376    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:35:26.538515 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:26.530526    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:26.538515 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:26.531763    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:26.915349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:26.877887    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:26.915349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:26.888860    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:35:26.929186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:26.925630    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:26.974340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:26.958514    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:26.974340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:26.961820    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:35:26.974340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:26.963678    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:35:27.265377 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.240729    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:27.275338 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.270492    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq" podUID="c1a40e94-c560-4f09-b721-533261b4ff25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:27.306369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.302447    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:35:27.330893 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.330866    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:35:27.332117 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.332096    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:35:27.333046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.333030    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:35:27.333212 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.333202    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:35:27.333325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.333315    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:35:27.333439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.333430    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:35:27.333540 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.333531    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:35:27.556570 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.556445    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:27.568388 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.568354    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:35:27.572323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.569140    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:27.580335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.573739    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:35:27.690950 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.690922    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:35:27.806319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.805731    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:35:27.893529 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.893491    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:27.987345 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.986915    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:27.987345 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.987249    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 31 13:35:27.999344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:27.992019    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:35:28.224917 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:28.224835    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:35:28.229343 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:28.226048    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:35:28.272094 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:28.271978    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq" podUID="c1a40e94-c560-4f09-b721-533261b4ff25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:28.291816 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:28.290860    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:28.291816 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:28.291412    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:35:28.570440 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:28.559022    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:35:28.602214 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:28.602183    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:35:28.624805 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:28.624411    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:28.694578 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:28.693608    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:28.708369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:28.707762    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:35:28.802373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:28.802327    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:28.811768 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:28.806367    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:29.107559 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:29.107511    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:29.241750 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:29.241702    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:29.282745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:29.281602    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq" podUID="c1a40e94-c560-4f09-b721-533261b4ff25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:29.298231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:29.297565    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:35:29.618329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:29.616769    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:35:29.649315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:29.648684    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:35:29.654314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:29.650909    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:29.711499 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:29.710136    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:35:33.997567 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:33.997479    4660 patch_prober.go:29] interesting pod/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:35:33.998067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:33.997564    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:35.266880 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:35.266838    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"]
Oct 31 13:35:35.267728 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:35.267665    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd" podUID="3d603cf6-4cc4-4159-a02f-1382522bf109" containerName="maas-api" containerID="cri-o://4ddf79e66d9ad6546925dfdaa706507668b9b1fd32d53229a051861307bed516" gracePeriod=30
Oct 31 13:35:36.009527 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:36.008701    4660 generic.go:356] "Generic (PLEG): container finished" podID="3d603cf6-4cc4-4159-a02f-1382522bf109" containerID="4ddf79e66d9ad6546925dfdaa706507668b9b1fd32d53229a051861307bed516" exitCode=0
Oct 31 13:35:36.009527 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:36.008762    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd" event={"ID":"3d603cf6-4cc4-4159-a02f-1382522bf109","Type":"ContainerDied","Data":"4ddf79e66d9ad6546925dfdaa706507668b9b1fd32d53229a051861307bed516"}
Oct 31 13:35:36.270913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:36.270835    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:35:36.363318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:36.360800    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d603cf6-4cc4-4159-a02f-1382522bf109-kube-api-access-vn4gm\" (UniqueName: \"kubernetes.io/projected/3d603cf6-4cc4-4159-a02f-1382522bf109-kube-api-access-vn4gm\") pod \"3d603cf6-4cc4-4159-a02f-1382522bf109\" (UID: \"3d603cf6-4cc4-4159-a02f-1382522bf109\") "
Oct 31 13:35:36.363318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:36.360853    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d603cf6-4cc4-4159-a02f-1382522bf109-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d603cf6-4cc4-4159-a02f-1382522bf109-metrics-certs\") pod \"3d603cf6-4cc4-4159-a02f-1382522bf109\" (UID: \"3d603cf6-4cc4-4159-a02f-1382522bf109\") "
Oct 31 13:35:36.363318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:36.360922    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d603cf6-4cc4-4159-a02f-1382522bf109-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d603cf6-4cc4-4159-a02f-1382522bf109-maas-api-tls\") pod \"3d603cf6-4cc4-4159-a02f-1382522bf109\" (UID: \"3d603cf6-4cc4-4159-a02f-1382522bf109\") "
Oct 31 13:35:36.363318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:36.361047    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d603cf6-4cc4-4159-a02f-1382522bf109-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d603cf6-4cc4-4159-a02f-1382522bf109-openshift-service-ca\") pod \"3d603cf6-4cc4-4159-a02f-1382522bf109\" (UID: \"3d603cf6-4cc4-4159-a02f-1382522bf109\") "
Oct 31 13:35:36.363318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:36.362212    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d603cf6-4cc4-4159-a02f-1382522bf109-openshift-service-ca" pod "3d603cf6-4cc4-4159-a02f-1382522bf109" (UID: "3d603cf6-4cc4-4159-a02f-1382522bf109"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:35:36.363318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:36.362588    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d603cf6-4cc4-4159-a02f-1382522bf109-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:36.393651 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:36.387432    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d603cf6-4cc4-4159-a02f-1382522bf109-metrics-certs" pod "3d603cf6-4cc4-4159-a02f-1382522bf109" (UID: "3d603cf6-4cc4-4159-a02f-1382522bf109"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:35:36.393651 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:36.391431    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d603cf6-4cc4-4159-a02f-1382522bf109-maas-api-tls" pod "3d603cf6-4cc4-4159-a02f-1382522bf109" (UID: "3d603cf6-4cc4-4159-a02f-1382522bf109"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:35:36.393651 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:36.392503    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d603cf6-4cc4-4159-a02f-1382522bf109-kube-api-access-vn4gm" pod "3d603cf6-4cc4-4159-a02f-1382522bf109" (UID: "3d603cf6-4cc4-4159-a02f-1382522bf109"). InnerVolumeSpecName "kube-api-access-vn4gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:35:36.464304 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:36.464268    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn4gm\" (UniqueName: \"kubernetes.io/projected/3d603cf6-4cc4-4159-a02f-1382522bf109-kube-api-access-vn4gm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:36.464506 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:36.464343    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d603cf6-4cc4-4159-a02f-1382522bf109-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:36.464506 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:36.464355    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d603cf6-4cc4-4159-a02f-1382522bf109-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:37.078317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:37.077613    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"
Oct 31 13:35:37.078317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:37.077779    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd" event={"ID":"3d603cf6-4cc4-4159-a02f-1382522bf109","Type":"ContainerDied","Data":"dd1d7d237fe3835fa9384d82fff55dcbf9f6997df47f7d2e927d85b3fbdee270"}
Oct 31 13:35:37.078317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:37.077855    4660 scope.go:122] "RemoveContainer" containerID="4ddf79e66d9ad6546925dfdaa706507668b9b1fd32d53229a051861307bed516"
Oct 31 13:35:37.160439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:37.151974    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"]
Oct 31 13:35:37.172351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:37.160982    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466002-7dbdd75659-6jfcd"]
Oct 31 13:35:38.571795 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:38.569155    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d603cf6-4cc4-4159-a02f-1382522bf109" path="/var/lib/kubelet/pods/3d603cf6-4cc4-4159-a02f-1382522bf109/volumes"
Oct 31 13:35:39.098703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:39.098668    4660 generic.go:356] "Generic (PLEG): container finished" podID="c5d5ff03-7667-4572-9447-e2972398e04e" containerID="3c3f876906cb56e137fa72e5d512b21b4e1bc9cedcadac824c6705e35f9ded66" exitCode=0
Oct 31 13:35:39.098880 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:39.098743    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" event={"ID":"c5d5ff03-7667-4572-9447-e2972398e04e","Type":"ContainerDied","Data":"3c3f876906cb56e137fa72e5d512b21b4e1bc9cedcadac824c6705e35f9ded66"}
Oct 31 13:35:39.099144 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:39.099131    4660 scope.go:122] "RemoveContainer" containerID="3c3f876906cb56e137fa72e5d512b21b4e1bc9cedcadac824c6705e35f9ded66"
Oct 31 13:35:39.100793 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:39.100761    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:35:39.100906 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:39.100853    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:35:39.100906 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:39.100902    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:35:40.151812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:40.149101    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" event={"ID":"c5d5ff03-7667-4572-9447-e2972398e04e","Type":"ContainerStarted","Data":"ec288aa507f384ec02bd94907cb655242611bc2827f8f5beffe333894719f204"}
Oct 31 13:35:40.209325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:40.194780    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:35:40.209325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:40.194914    4660 patch_prober.go:29] interesting pod/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 13:35:40.209325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:40.194955    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 13:35:41.195174 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:41.194398    4660 patch_prober.go:29] interesting pod/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 13:35:41.195174 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:41.194473    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 13:35:42.194835 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:42.194779    4660 patch_prober.go:29] interesting pod/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 13:35:42.195112 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:42.194868    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 13:35:43.194371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.194335    4660 patch_prober.go:29] interesting pod/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 13:35:43.194768 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.194398    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 13:35:43.563939 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.563849    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"]
Oct 31 13:35:43.564619 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.564602    4660 cpu_manager.go:414] "containerMap: removing container" podUID="3d603cf6-4cc4-4159-a02f-1382522bf109" containerName="maas-api"
Oct 31 13:35:43.564765 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.564747    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d603cf6-4cc4-4159-a02f-1382522bf109" containerName="maas-api"
Oct 31 13:35:43.565066 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.565051    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d603cf6-4cc4-4159-a02f-1382522bf109" containerName="maas-api"
Oct 31 13:35:43.568858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.568841    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:35:43.583983 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.583953    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"]
Oct 31 13:35:43.678322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.674090    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6aa924-serving-cert\""
Oct 31 13:35:43.678322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.675839    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6bae42a-e94c-453a-8a9c-9da54f465a6c-metrics-certs\") pod \"maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr\" (UID: \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:35:43.678322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.675892    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vlql\" (UniqueName: \"kubernetes.io/projected/c6bae42a-e94c-453a-8a9c-9da54f465a6c-kube-api-access-6vlql\") pod \"maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr\" (UID: \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:35:43.678322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.676037    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6bae42a-e94c-453a-8a9c-9da54f465a6c-maas-api-tls\") pod \"maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr\" (UID: \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:35:43.678322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.676279    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6bae42a-e94c-453a-8a9c-9da54f465a6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr\" (UID: \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:35:43.785394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.779541    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6bae42a-e94c-453a-8a9c-9da54f465a6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr\" (UID: \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:35:43.785394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.779639    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6bae42a-e94c-453a-8a9c-9da54f465a6c-metrics-certs\") pod \"maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr\" (UID: \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:35:43.785394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.779683    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vlql\" (UniqueName: \"kubernetes.io/projected/c6bae42a-e94c-453a-8a9c-9da54f465a6c-kube-api-access-6vlql\") pod \"maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr\" (UID: \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:35:43.785394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.779847    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6bae42a-e94c-453a-8a9c-9da54f465a6c-maas-api-tls\") pod \"maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr\" (UID: \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:35:43.785394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.782179    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6bae42a-e94c-453a-8a9c-9da54f465a6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr\" (UID: \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:35:43.785394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.785128    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6bae42a-e94c-453a-8a9c-9da54f465a6c-maas-api-tls\") pod \"maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr\" (UID: \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:35:43.793477 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.793418    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6bae42a-e94c-453a-8a9c-9da54f465a6c-metrics-certs\") pod \"maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr\" (UID: \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:35:43.795580 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.795538    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vlql\" (UniqueName: \"kubernetes.io/projected/c6bae42a-e94c-453a-8a9c-9da54f465a6c-kube-api-access-6vlql\") pod \"maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr\" (UID: \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:35:43.897552 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:43.897512    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:35:44.435616 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:44.403955    4660 patch_prober.go:29] interesting pod/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 13:35:44.435616 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:44.404048    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 13:35:44.865667 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:44.863841    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"]
Oct 31 13:35:44.865667 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:44.864596    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c" podUID="1d7ee133-8731-4bc3-bfa0-157935a79a24" containerName="istio-proxy" containerID="cri-o://f8136024f75f1426265ee27d8e5c5915b601b975a93d84f09330a30a939a1021" gracePeriod=30
Oct 31 13:35:45.197317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:45.196139    4660 patch_prober.go:29] interesting pod/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 13:35:45.197317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:45.196209    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 13:35:45.284357 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:45.284325    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"]
Oct 31 13:35:45.288599 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:45.288575    4660 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:35:45.553267 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:45.553182    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr" event={"ID":"c6bae42a-e94c-453a-8a9c-9da54f465a6c","Type":"ContainerStarted","Data":"f5c6cb863edc1cd6be9cbb98c955a87931bd297d6649b78137028771f5146ff0"}
Oct 31 13:35:46.028456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:46.028405    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11110-ea2272"]
Oct 31 13:35:46.049325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:46.035862    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11110-ea2272"
Oct 31 13:35:46.049325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:46.046047    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-whw9x\""
Oct 31 13:35:46.056525 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:46.055947    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2l9v\" (UniqueName: \"kubernetes.io/projected/a62629e6-5f44-4fbb-8d42-7c931259cc62-kube-api-access-b2l9v\") pod \"test-curl-11110-ea2272\" (UID: \"a62629e6-5f44-4fbb-8d42-7c931259cc62\") " pod="opendatahub/test-curl-11110-ea2272"
Oct 31 13:35:46.073946 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:46.073877    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11110-ea2272"]
Oct 31 13:35:46.158488 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:46.158396    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2l9v\" (UniqueName: \"kubernetes.io/projected/a62629e6-5f44-4fbb-8d42-7c931259cc62-kube-api-access-b2l9v\") pod \"test-curl-11110-ea2272\" (UID: \"a62629e6-5f44-4fbb-8d42-7c931259cc62\") " pod="opendatahub/test-curl-11110-ea2272"
Oct 31 13:35:46.174777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:46.174747    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2l9v\" (UniqueName: \"kubernetes.io/projected/a62629e6-5f44-4fbb-8d42-7c931259cc62-kube-api-access-b2l9v\") pod \"test-curl-11110-ea2272\" (UID: \"a62629e6-5f44-4fbb-8d42-7c931259cc62\") " pod="opendatahub/test-curl-11110-ea2272"
Oct 31 13:35:46.194495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:46.194460    4660 patch_prober.go:29] interesting pod/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 13:35:46.194635 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:46.194510    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 13:35:46.374135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:46.374099    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11110-ea2272"
Oct 31 13:35:46.562927 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:46.562887    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr" event={"ID":"c6bae42a-e94c-453a-8a9c-9da54f465a6c","Type":"ContainerStarted","Data":"f5c2dde697df00d8c4962b395540d3e0b097f128c4aca392fc58f7ab302d7a39"}
Oct 31 13:35:46.563349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:46.562952    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:35:46.589193 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:46.589146    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr" podStartSLOduration=2.990906113 podStartE2EDuration="3.589133008s" totalImagesPullingTime="598.226895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:46.585904479 +0000 UTC m=+1946.204563645" watchObservedRunningTime="2026-10-31 13:35:46.589133008 +0000 UTC m=+1946.207792171"
Oct 31 13:35:46.671167 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:35:46.671132    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda62629e6_5f44_4fbb_8d42_7c931259cc62.slice/crio-50eb710e1291644d14de3e58eecd820f9785cffc52ceeb3a4a8fd52040570cbd WatchSource:0}: Error finding container 50eb710e1291644d14de3e58eecd820f9785cffc52ceeb3a4a8fd52040570cbd: Status 404 returned error can't find the container with id 50eb710e1291644d14de3e58eecd820f9785cffc52ceeb3a4a8fd52040570cbd
Oct 31 13:35:46.671948 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:46.671924    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11110-ea2272"]
Oct 31 13:35:47.199800 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:47.199769    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:35:47.200288 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:47.200270    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:35:47.201033 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:47.201016    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:35:47.575028 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:47.574937    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11110-ea2272" event={"ID":"a62629e6-5f44-4fbb-8d42-7c931259cc62","Type":"ContainerStarted","Data":"a0bbb120aa9c045826956b5ac2f038a730df99fa4158df17aad537042ac07dc4"}
Oct 31 13:35:47.575028 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:47.574983    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11110-ea2272" event={"ID":"a62629e6-5f44-4fbb-8d42-7c931259cc62","Type":"ContainerStarted","Data":"50eb710e1291644d14de3e58eecd820f9785cffc52ceeb3a4a8fd52040570cbd"}
Oct 31 13:35:47.595099 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:47.595052    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11110-ea2272" podStartSLOduration=2.595037949 podStartE2EDuration="2.595037949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:47.592857106 +0000 UTC m=+1947.211516269" watchObservedRunningTime="2026-10-31 13:35:47.595037949 +0000 UTC m=+1947.213697113"
Oct 31 13:35:48.346273 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:48.346238    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11110-ea2272"]
Oct 31 13:35:48.354220 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:48.354187    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11110-ea2272"]
Oct 31 13:35:49.606363 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:49.606324    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11110-ea2272" podUID="a62629e6-5f44-4fbb-8d42-7c931259cc62" containerName="test-curl-11110-ea2272" containerID="cri-o://a0bbb120aa9c045826956b5ac2f038a730df99fa4158df17aad537042ac07dc4" gracePeriod=2
Oct 31 13:35:50.225280 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.225248    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:35:50.359919 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.359882    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-data\") pod \"1d7ee133-8731-4bc3-bfa0-157935a79a24\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") "
Oct 31 13:35:50.360106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.360035    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-token\" (UniqueName: \"kubernetes.io/projected/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-token\") pod \"1d7ee133-8731-4bc3-bfa0-157935a79a24\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") "
Oct 31 13:35:50.360106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.360071    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d7ee133-8731-4bc3-bfa0-157935a79a24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d7ee133-8731-4bc3-bfa0-157935a79a24-istiod-ca-cert\") pod \"1d7ee133-8731-4bc3-bfa0-157935a79a24\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") "
Oct 31 13:35:50.360229 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.360126    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-data" pod "1d7ee133-8731-4bc3-bfa0-157935a79a24" (UID: "1d7ee133-8731-4bc3-bfa0-157935a79a24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:35:50.360229 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.360157    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-credential-socket\") pod \"1d7ee133-8731-4bc3-bfa0-157935a79a24\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") "
Oct 31 13:35:50.360229 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.360187    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7ee133-8731-4bc3-bfa0-157935a79a24-kube-api-access-5wsmh\" (UniqueName: \"kubernetes.io/projected/1d7ee133-8731-4bc3-bfa0-157935a79a24-kube-api-access-5wsmh\") pod \"1d7ee133-8731-4bc3-bfa0-157935a79a24\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") "
Oct 31 13:35:50.360444 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.360253    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-workload-certs\") pod \"1d7ee133-8731-4bc3-bfa0-157935a79a24\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") "
Oct 31 13:35:50.360444 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.360417    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-workload-socket\") pod \"1d7ee133-8731-4bc3-bfa0-157935a79a24\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") "
Oct 31 13:35:50.360544 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.360442    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-credential-socket" pod "1d7ee133-8731-4bc3-bfa0-157935a79a24" (UID: "1d7ee133-8731-4bc3-bfa0-157935a79a24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:35:50.360594 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.360545    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-envoy\") pod \"1d7ee133-8731-4bc3-bfa0-157935a79a24\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") "
Oct 31 13:35:50.360594 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.360553    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7ee133-8731-4bc3-bfa0-157935a79a24-istiod-ca-cert" pod "1d7ee133-8731-4bc3-bfa0-157935a79a24" (UID: "1d7ee133-8731-4bc3-bfa0-157935a79a24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:35:50.360697 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.360590    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-workload-certs" pod "1d7ee133-8731-4bc3-bfa0-157935a79a24" (UID: "1d7ee133-8731-4bc3-bfa0-157935a79a24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:35:50.360697 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.360616    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-podinfo\") pod \"1d7ee133-8731-4bc3-bfa0-157935a79a24\" (UID: \"1d7ee133-8731-4bc3-bfa0-157935a79a24\") "
Oct 31 13:35:50.360801 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.360708    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-workload-socket" pod "1d7ee133-8731-4bc3-bfa0-157935a79a24" (UID: "1d7ee133-8731-4bc3-bfa0-157935a79a24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:35:50.361176 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.361137    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:50.361176 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.361162    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:50.361176 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.361176    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:50.361365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.361189    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:50.361365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.361201    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d7ee133-8731-4bc3-bfa0-157935a79a24-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:50.362397 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.362375    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-token" pod "1d7ee133-8731-4bc3-bfa0-157935a79a24" (UID: "1d7ee133-8731-4bc3-bfa0-157935a79a24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:35:50.362607 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.362587    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-podinfo" pod "1d7ee133-8731-4bc3-bfa0-157935a79a24" (UID: "1d7ee133-8731-4bc3-bfa0-157935a79a24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:35:50.362813 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.362796    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-envoy" pod "1d7ee133-8731-4bc3-bfa0-157935a79a24" (UID: "1d7ee133-8731-4bc3-bfa0-157935a79a24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:35:50.362813 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.362806    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7ee133-8731-4bc3-bfa0-157935a79a24-kube-api-access-5wsmh" pod "1d7ee133-8731-4bc3-bfa0-157935a79a24" (UID: "1d7ee133-8731-4bc3-bfa0-157935a79a24"). InnerVolumeSpecName "kube-api-access-5wsmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:35:50.463169 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.463094    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:50.463169 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.463124    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wsmh\" (UniqueName: \"kubernetes.io/projected/1d7ee133-8731-4bc3-bfa0-157935a79a24-kube-api-access-5wsmh\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:50.463169 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.463137    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:50.463169 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.463145    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d7ee133-8731-4bc3-bfa0-157935a79a24-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:50.616722 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.616678    4660 generic.go:356] "Generic (PLEG): container finished" podID="1d7ee133-8731-4bc3-bfa0-157935a79a24" containerID="f8136024f75f1426265ee27d8e5c5915b601b975a93d84f09330a30a939a1021" exitCode=0
Oct 31 13:35:50.617125 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.616733    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"
Oct 31 13:35:50.617125 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.616769    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c" event={"ID":"1d7ee133-8731-4bc3-bfa0-157935a79a24","Type":"ContainerDied","Data":"f8136024f75f1426265ee27d8e5c5915b601b975a93d84f09330a30a939a1021"}
Oct 31 13:35:50.617125 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.616838    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c" event={"ID":"1d7ee133-8731-4bc3-bfa0-157935a79a24","Type":"ContainerDied","Data":"3c63080d065bb8a87c051531d989ca1c32c07b6f69b078a73474d8494ed00c5c"}
Oct 31 13:35:50.617125 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.616869    4660 scope.go:122] "RemoveContainer" containerID="f8136024f75f1426265ee27d8e5c5915b601b975a93d84f09330a30a939a1021"
Oct 31 13:35:50.635439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.632495    4660 scope.go:122] "RemoveContainer" containerID="f8136024f75f1426265ee27d8e5c5915b601b975a93d84f09330a30a939a1021"
Oct 31 13:35:50.635439 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:35:50.633949    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8136024f75f1426265ee27d8e5c5915b601b975a93d84f09330a30a939a1021\": container with ID starting with f8136024f75f1426265ee27d8e5c5915b601b975a93d84f09330a30a939a1021 not found: ID does not exist" containerID="f8136024f75f1426265ee27d8e5c5915b601b975a93d84f09330a30a939a1021"
Oct 31 13:35:50.635439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.633991    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8136024f75f1426265ee27d8e5c5915b601b975a93d84f09330a30a939a1021"} err="failed to get container status \"f8136024f75f1426265ee27d8e5c5915b601b975a93d84f09330a30a939a1021\": rpc error: code = NotFound desc = could not find container \"f8136024f75f1426265ee27d8e5c5915b601b975a93d84f09330a30a939a1021\": container with ID starting with f8136024f75f1426265ee27d8e5c5915b601b975a93d84f09330a30a939a1021 not found: ID does not exist"
Oct 31 13:35:50.651072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.651042    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"]
Oct 31 13:35:50.656031 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:50.656006    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5dj6c"]
Oct 31 13:35:50.735341 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:35:50.735215    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:35:51.863695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:51.863665    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11110-ea2272"
Oct 31 13:35:51.987209 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:51.987175    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62629e6-5f44-4fbb-8d42-7c931259cc62-kube-api-access-b2l9v\" (UniqueName: \"kubernetes.io/projected/a62629e6-5f44-4fbb-8d42-7c931259cc62-kube-api-access-b2l9v\") pod \"a62629e6-5f44-4fbb-8d42-7c931259cc62\" (UID: \"a62629e6-5f44-4fbb-8d42-7c931259cc62\") "
Oct 31 13:35:51.989703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:51.989670    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62629e6-5f44-4fbb-8d42-7c931259cc62-kube-api-access-b2l9v" pod "a62629e6-5f44-4fbb-8d42-7c931259cc62" (UID: "a62629e6-5f44-4fbb-8d42-7c931259cc62"). InnerVolumeSpecName "kube-api-access-b2l9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:35:52.089177 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:52.089144    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2l9v\" (UniqueName: \"kubernetes.io/projected/a62629e6-5f44-4fbb-8d42-7c931259cc62-kube-api-access-b2l9v\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:52.528969 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:52.528935    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d7ee133-8731-4bc3-bfa0-157935a79a24" path="/var/lib/kubelet/pods/1d7ee133-8731-4bc3-bfa0-157935a79a24/volumes"
Oct 31 13:35:52.529418 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:52.529401    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62629e6-5f44-4fbb-8d42-7c931259cc62" path="/var/lib/kubelet/pods/a62629e6-5f44-4fbb-8d42-7c931259cc62/volumes"
Oct 31 13:35:52.580891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:52.580863    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:35:52.651373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:52.645273    4660 generic.go:356] "Generic (PLEG): container finished" podID="a62629e6-5f44-4fbb-8d42-7c931259cc62" containerID="a0bbb120aa9c045826956b5ac2f038a730df99fa4158df17aad537042ac07dc4" exitCode=137
Oct 31 13:35:52.651373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:52.645405    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11110-ea2272"
Oct 31 13:35:52.651373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:52.645519    4660 scope.go:122] "RemoveContainer" containerID="a0bbb120aa9c045826956b5ac2f038a730df99fa4158df17aad537042ac07dc4"
Oct 31 13:35:52.661761 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:52.661732    4660 scope.go:122] "RemoveContainer" containerID="a0bbb120aa9c045826956b5ac2f038a730df99fa4158df17aad537042ac07dc4"
Oct 31 13:35:52.662064 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:35:52.662044    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0bbb120aa9c045826956b5ac2f038a730df99fa4158df17aad537042ac07dc4\": container with ID starting with a0bbb120aa9c045826956b5ac2f038a730df99fa4158df17aad537042ac07dc4 not found: ID does not exist" containerID="a0bbb120aa9c045826956b5ac2f038a730df99fa4158df17aad537042ac07dc4"
Oct 31 13:35:52.662151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:52.662077    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bbb120aa9c045826956b5ac2f038a730df99fa4158df17aad537042ac07dc4"} err="failed to get container status \"a0bbb120aa9c045826956b5ac2f038a730df99fa4158df17aad537042ac07dc4\": rpc error: code = NotFound desc = could not find container \"a0bbb120aa9c045826956b5ac2f038a730df99fa4158df17aad537042ac07dc4\": container with ID starting with a0bbb120aa9c045826956b5ac2f038a730df99fa4158df17aad537042ac07dc4 not found: ID does not exist"
Oct 31 13:35:55.586953 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.586898    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"]
Oct 31 13:35:55.601728 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.601689    4660 cpu_manager.go:414] "containerMap: removing container" podUID="1d7ee133-8731-4bc3-bfa0-157935a79a24" containerName="istio-proxy"
Oct 31 13:35:55.604280 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.604252    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7ee133-8731-4bc3-bfa0-157935a79a24" containerName="istio-proxy"
Oct 31 13:35:55.604540 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.604526    4660 cpu_manager.go:414] "containerMap: removing container" podUID="a62629e6-5f44-4fbb-8d42-7c931259cc62" containerName="test-curl-11110-ea2272"
Oct 31 13:35:55.604621 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.604610    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62629e6-5f44-4fbb-8d42-7c931259cc62" containerName="test-curl-11110-ea2272"
Oct 31 13:35:55.604918 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.604905    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62629e6-5f44-4fbb-8d42-7c931259cc62" containerName="test-curl-11110-ea2272"
Oct 31 13:35:55.605028 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.605018    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7ee133-8731-4bc3-bfa0-157935a79a24" containerName="istio-proxy"
Oct 31 13:35:55.632017 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.631986    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"]
Oct 31 13:35:55.632164 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.632111    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 13:35:55.636866 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.636838    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-15dc2f-serving-cert\""
Oct 31 13:35:55.731328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.730924    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533c7ae9-3ae7-4cd4-a38f-98c35110b250-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw\" (UID: \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 13:35:55.731328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.731043    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltvvd\" (UniqueName: \"kubernetes.io/projected/533c7ae9-3ae7-4cd4-a38f-98c35110b250-kube-api-access-ltvvd\") pod \"maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw\" (UID: \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 13:35:55.731328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.731118    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533c7ae9-3ae7-4cd4-a38f-98c35110b250-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw\" (UID: \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 13:35:55.731627 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.731436    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533c7ae9-3ae7-4cd4-a38f-98c35110b250-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw\" (UID: \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 13:35:55.833511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.833470    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltvvd\" (UniqueName: \"kubernetes.io/projected/533c7ae9-3ae7-4cd4-a38f-98c35110b250-kube-api-access-ltvvd\") pod \"maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw\" (UID: \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 13:35:55.834129 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.833566    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533c7ae9-3ae7-4cd4-a38f-98c35110b250-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw\" (UID: \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 13:35:55.834222 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.834139    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533c7ae9-3ae7-4cd4-a38f-98c35110b250-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw\" (UID: \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 13:35:55.834276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.834225    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533c7ae9-3ae7-4cd4-a38f-98c35110b250-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw\" (UID: \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 13:35:55.835149 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.835121    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533c7ae9-3ae7-4cd4-a38f-98c35110b250-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw\" (UID: \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 13:35:55.837603 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.837539    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533c7ae9-3ae7-4cd4-a38f-98c35110b250-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw\" (UID: \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 13:35:55.838729 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.838700    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533c7ae9-3ae7-4cd4-a38f-98c35110b250-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw\" (UID: \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 13:35:55.846923 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.846902    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltvvd\" (UniqueName: \"kubernetes.io/projected/533c7ae9-3ae7-4cd4-a38f-98c35110b250-kube-api-access-ltvvd\") pod \"maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw\" (UID: \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 13:35:55.957937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:55.957898    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 13:35:56.357971 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:56.357941    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"]
Oct 31 13:35:56.693151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:56.693117    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw" event={"ID":"533c7ae9-3ae7-4cd4-a38f-98c35110b250","Type":"ContainerStarted","Data":"681ce368deb8e291b685c1c1b719891621b6cab7ce3aa3923949098bf2d3aa84"}
Oct 31 13:35:57.704615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:57.704577    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw" event={"ID":"533c7ae9-3ae7-4cd4-a38f-98c35110b250","Type":"ContainerStarted","Data":"61222bc638035d0b17b0786b616c72fabbadde58638182cc0be00cbe74e85066"}
Oct 31 13:35:57.705024 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:57.704644    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 13:35:57.748371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:35:57.748302    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw" podStartSLOduration=2.27069333 podStartE2EDuration="2.748268803s" totalImagesPullingTime="477.575473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:57.741970698 +0000 UTC m=+1957.360629867" watchObservedRunningTime="2026-10-31 13:35:57.748268803 +0000 UTC m=+1957.366927967"
Oct 31 13:36:00.449319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:00.426512    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"]
Oct 31 13:36:00.449319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:00.427158    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt" podUID="53bbd017-4f45-4cfb-9d13-e50b750a2590" containerName="maas-api" containerID="cri-o://aa041fcaf49c4f909304b9fa18fa6322caef715a8303ef79993e8166b8eb2525" gracePeriod=30
Oct 31 13:36:01.060729 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:01.060455    4660 generic.go:356] "Generic (PLEG): container finished" podID="53bbd017-4f45-4cfb-9d13-e50b750a2590" containerID="aa041fcaf49c4f909304b9fa18fa6322caef715a8303ef79993e8166b8eb2525" exitCode=0
Oct 31 13:36:01.060729 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:01.060629    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt" event={"ID":"53bbd017-4f45-4cfb-9d13-e50b750a2590","Type":"ContainerDied","Data":"aa041fcaf49c4f909304b9fa18fa6322caef715a8303ef79993e8166b8eb2525"}
Oct 31 13:36:01.125364 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:01.122072    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:36:01.289504 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:01.289419    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53bbd017-4f45-4cfb-9d13-e50b750a2590-metrics-certs\" (UniqueName: \"kubernetes.io/secret/53bbd017-4f45-4cfb-9d13-e50b750a2590-metrics-certs\") pod \"53bbd017-4f45-4cfb-9d13-e50b750a2590\" (UID: \"53bbd017-4f45-4cfb-9d13-e50b750a2590\") "
Oct 31 13:36:01.289686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:01.289506    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53bbd017-4f45-4cfb-9d13-e50b750a2590-kube-api-access-2jl6n\" (UniqueName: \"kubernetes.io/projected/53bbd017-4f45-4cfb-9d13-e50b750a2590-kube-api-access-2jl6n\") pod \"53bbd017-4f45-4cfb-9d13-e50b750a2590\" (UID: \"53bbd017-4f45-4cfb-9d13-e50b750a2590\") "
Oct 31 13:36:01.289686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:01.289624    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53bbd017-4f45-4cfb-9d13-e50b750a2590-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bbd017-4f45-4cfb-9d13-e50b750a2590-openshift-service-ca\") pod \"53bbd017-4f45-4cfb-9d13-e50b750a2590\" (UID: \"53bbd017-4f45-4cfb-9d13-e50b750a2590\") "
Oct 31 13:36:01.289860 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:01.289840    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53bbd017-4f45-4cfb-9d13-e50b750a2590-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53bbd017-4f45-4cfb-9d13-e50b750a2590-maas-api-tls\") pod \"53bbd017-4f45-4cfb-9d13-e50b750a2590\" (UID: \"53bbd017-4f45-4cfb-9d13-e50b750a2590\") "
Oct 31 13:36:01.290064 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:01.290038    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bbd017-4f45-4cfb-9d13-e50b750a2590-openshift-service-ca" pod "53bbd017-4f45-4cfb-9d13-e50b750a2590" (UID: "53bbd017-4f45-4cfb-9d13-e50b750a2590"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:01.290679 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:01.290657    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bbd017-4f45-4cfb-9d13-e50b750a2590-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:01.294785 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:01.294753    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bbd017-4f45-4cfb-9d13-e50b750a2590-kube-api-access-2jl6n" pod "53bbd017-4f45-4cfb-9d13-e50b750a2590" (UID: "53bbd017-4f45-4cfb-9d13-e50b750a2590"). InnerVolumeSpecName "kube-api-access-2jl6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:01.295030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:01.294999    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53bbd017-4f45-4cfb-9d13-e50b750a2590-maas-api-tls" pod "53bbd017-4f45-4cfb-9d13-e50b750a2590" (UID: "53bbd017-4f45-4cfb-9d13-e50b750a2590"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:36:01.295030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:01.295012    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53bbd017-4f45-4cfb-9d13-e50b750a2590-metrics-certs" pod "53bbd017-4f45-4cfb-9d13-e50b750a2590" (UID: "53bbd017-4f45-4cfb-9d13-e50b750a2590"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:36:01.391861 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:01.391831    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53bbd017-4f45-4cfb-9d13-e50b750a2590-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:01.391861 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:01.391858    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53bbd017-4f45-4cfb-9d13-e50b750a2590-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:01.391861 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:01.391871    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jl6n\" (UniqueName: \"kubernetes.io/projected/53bbd017-4f45-4cfb-9d13-e50b750a2590-kube-api-access-2jl6n\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:02.070454 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:02.070424    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"
Oct 31 13:36:02.070918 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:02.070453    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt" event={"ID":"53bbd017-4f45-4cfb-9d13-e50b750a2590","Type":"ContainerDied","Data":"4d54a3823c7b2217e3795d43e3c483c4e24a229daa7598e86815569107fb68d4"}
Oct 31 13:36:02.070918 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:02.070512    4660 scope.go:122] "RemoveContainer" containerID="aa041fcaf49c4f909304b9fa18fa6322caef715a8303ef79993e8166b8eb2525"
Oct 31 13:36:02.108621 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:02.108595    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"]
Oct 31 13:36:02.119280 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:02.119250    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75c956-75f5dd6b9d-2fmkt"]
Oct 31 13:36:02.530371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:02.530340    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53bbd017-4f45-4cfb-9d13-e50b750a2590" path="/var/lib/kubelet/pods/53bbd017-4f45-4cfb-9d13-e50b750a2590/volumes"
Oct 31 13:36:03.739788 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:03.739758    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 13:36:06.041143 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.041101    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"]
Oct 31 13:36:06.142968 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.142925    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"]
Oct 31 13:36:06.144126 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.144094    4660 cpu_manager.go:414] "containerMap: removing container" podUID="53bbd017-4f45-4cfb-9d13-e50b750a2590" containerName="maas-api"
Oct 31 13:36:06.144126 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.144125    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53bbd017-4f45-4cfb-9d13-e50b750a2590" containerName="maas-api"
Oct 31 13:36:06.145440 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.145417    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="53bbd017-4f45-4cfb-9d13-e50b750a2590" containerName="maas-api"
Oct 31 13:36:06.152477 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.152449    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.157508 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.157480    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c45a37-openshift-default-dockercfg-xfwrm\""
Oct 31 13:36:06.168143 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.168116    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"]
Oct 31 13:36:06.196795 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.196753    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-workload-socket\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.197067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.197042    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-envoy\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.197590 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.197539    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-credential-socket\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.197769 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.197749    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4mj4\" (UniqueName: \"kubernetes.io/projected/30b88da6-a8a1-4882-8264-3ae18b1350a9-kube-api-access-s4mj4\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.197967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.197949    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-token\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.198133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.198106    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-data\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.198499 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.198142    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-podinfo\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.198499 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.198174    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b88da6-a8a1-4882-8264-3ae18b1350a9-istiod-ca-cert\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.198499 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.198337    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-workload-certs\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.302451 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.302349    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-token\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.302729 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.302710    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-data\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.302844 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.302829    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-podinfo\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.302943 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.302931    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b88da6-a8a1-4882-8264-3ae18b1350a9-istiod-ca-cert\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.303128 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.303114    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-workload-certs\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.303317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.303283    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-workload-socket\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.303431 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.303418    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-envoy\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.303705 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.303692    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-credential-socket\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.303803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.303789    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4mj4\" (UniqueName: \"kubernetes.io/projected/30b88da6-a8a1-4882-8264-3ae18b1350a9-kube-api-access-s4mj4\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.310341 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.304855    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-data\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.315130 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.315060    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b88da6-a8a1-4882-8264-3ae18b1350a9-istiod-ca-cert\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.319440 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.319411    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-workload-socket\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.341154 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.340850    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-credential-socket\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.341154 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.341109    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-workload-certs\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.377587 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.377511    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4mj4\" (UniqueName: \"kubernetes.io/projected/30b88da6-a8a1-4882-8264-3ae18b1350a9-kube-api-access-s4mj4\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.399600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.392924    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-token\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.399600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.393528    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-podinfo\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.401132 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.401059    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-envoy\") pod \"e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") " pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.482328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.479608    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:06.975701 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.975671    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"]
Oct 31 13:36:06.976772 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:36:06.976746    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b88da6_a8a1_4882_8264_3ae18b1350a9.slice/crio-4e855d0acd8ba05065b86948473b0018c8b36ad9553bf8ee75f9636a12d5e19f WatchSource:0}: Error finding container 4e855d0acd8ba05065b86948473b0018c8b36ad9553bf8ee75f9636a12d5e19f: Status 404 returned error can't find the container with id 4e855d0acd8ba05065b86948473b0018c8b36ad9553bf8ee75f9636a12d5e19f
Oct 31 13:36:06.980574 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.980541    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:06.980657 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.980611    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:06.980657 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:06.980641    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:07.164028 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:07.163981    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh" event={"ID":"30b88da6-a8a1-4882-8264-3ae18b1350a9","Type":"ContainerStarted","Data":"ad3f9d5af33a976432799429a354c069336a507eb64096de4eda3117c922e778"}
Oct 31 13:36:07.164498 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:07.164096    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh" event={"ID":"30b88da6-a8a1-4882-8264-3ae18b1350a9","Type":"ContainerStarted","Data":"4e855d0acd8ba05065b86948473b0018c8b36ad9553bf8ee75f9636a12d5e19f"}
Oct 31 13:36:07.200158 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:07.199689    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh" podStartSLOduration=1.199668113 podStartE2EDuration="1.199668113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:07.194117142 +0000 UTC m=+1966.812776366" watchObservedRunningTime="2026-10-31 13:36:07.199668113 +0000 UTC m=+1966.818327279"
Oct 31 13:36:07.483375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:07.483334    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:08.490093 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:08.490065    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:09.185786 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:09.185750    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:09.186825 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:09.186802    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:36:20.894086 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:36:20.894040    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:36:21.065579 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.065508    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq" podUID="c1a40e94-c560-4f09-b721-533261b4ff25" containerName="main" containerID="cri-o://82dade06584e27c60fa6f915cd4a559fa6eff51d147214467bf04661c46f04f0" gracePeriod=45
Oct 31 13:36:21.273841 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.273795    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq" podUID="c1a40e94-c560-4f09-b721-533261b4ff25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": dial tcp 10.133.0.103:8000: connect: connection refused"
Oct 31 13:36:21.613985 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.613628    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:36:21.699884 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.699845    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1a40e94-c560-4f09-b721-533261b4ff25-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a40e94-c560-4f09-b721-533261b4ff25-tls-certs\") pod \"c1a40e94-c560-4f09-b721-533261b4ff25\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") "
Oct 31 13:36:21.719011 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.718974    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-model-cache\") pod \"c1a40e94-c560-4f09-b721-533261b4ff25\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") "
Oct 31 13:36:21.721750 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.721621    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-home" pod "c1a40e94-c560-4f09-b721-533261b4ff25" (UID: "c1a40e94-c560-4f09-b721-533261b4ff25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.721750 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.721694    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-model-cache" pod "c1a40e94-c560-4f09-b721-533261b4ff25" (UID: "c1a40e94-c560-4f09-b721-533261b4ff25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.721750 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.721732    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-home\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-home\") pod \"c1a40e94-c560-4f09-b721-533261b4ff25\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") "
Oct 31 13:36:21.722041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.722022    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-dshm\") pod \"c1a40e94-c560-4f09-b721-533261b4ff25\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") "
Oct 31 13:36:21.722174 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.722160    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-tmp-dir\") pod \"c1a40e94-c560-4f09-b721-533261b4ff25\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") "
Oct 31 13:36:21.722315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.722284    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1a40e94-c560-4f09-b721-533261b4ff25-kube-api-access-vqmgw\" (UniqueName: \"kubernetes.io/projected/c1a40e94-c560-4f09-b721-533261b4ff25-kube-api-access-vqmgw\") pod \"c1a40e94-c560-4f09-b721-533261b4ff25\" (UID: \"c1a40e94-c560-4f09-b721-533261b4ff25\") "
Oct 31 13:36:21.723492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.723465    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.723621 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.723496    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.723778 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.723689    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-tmp-dir" pod "c1a40e94-c560-4f09-b721-533261b4ff25" (UID: "c1a40e94-c560-4f09-b721-533261b4ff25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.738737 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.738697    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a40e94-c560-4f09-b721-533261b4ff25-kube-api-access-vqmgw" pod "c1a40e94-c560-4f09-b721-533261b4ff25" (UID: "c1a40e94-c560-4f09-b721-533261b4ff25"). InnerVolumeSpecName "kube-api-access-vqmgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.738936 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.738911    4660 generic.go:356] "Generic (PLEG): container finished" podID="c1a40e94-c560-4f09-b721-533261b4ff25" containerID="82dade06584e27c60fa6f915cd4a559fa6eff51d147214467bf04661c46f04f0" exitCode=0
Oct 31 13:36:21.739013 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.738992    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq" event={"ID":"c1a40e94-c560-4f09-b721-533261b4ff25","Type":"ContainerDied","Data":"82dade06584e27c60fa6f915cd4a559fa6eff51d147214467bf04661c46f04f0"}
Oct 31 13:36:21.739072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.739031    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq" event={"ID":"c1a40e94-c560-4f09-b721-533261b4ff25","Type":"ContainerDied","Data":"83abc515e54f66fb82ee050d57928d47b0227e1c687dbfa26c535907521707a4"}
Oct 31 13:36:21.739072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.739055    4660 scope.go:122] "RemoveContainer" containerID="82dade06584e27c60fa6f915cd4a559fa6eff51d147214467bf04661c46f04f0"
Oct 31 13:36:21.739239 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.739224    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"
Oct 31 13:36:21.741569 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.741531    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-dshm" pod "c1a40e94-c560-4f09-b721-533261b4ff25" (UID: "c1a40e94-c560-4f09-b721-533261b4ff25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.750333 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.746416    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a40e94-c560-4f09-b721-533261b4ff25-tls-certs" pod "c1a40e94-c560-4f09-b721-533261b4ff25" (UID: "c1a40e94-c560-4f09-b721-533261b4ff25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:36:21.780124 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.780014    4660 scope.go:122] "RemoveContainer" containerID="82dade06584e27c60fa6f915cd4a559fa6eff51d147214467bf04661c46f04f0"
Oct 31 13:36:21.780575 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:36:21.780440    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82dade06584e27c60fa6f915cd4a559fa6eff51d147214467bf04661c46f04f0\": container with ID starting with 82dade06584e27c60fa6f915cd4a559fa6eff51d147214467bf04661c46f04f0 not found: ID does not exist" containerID="82dade06584e27c60fa6f915cd4a559fa6eff51d147214467bf04661c46f04f0"
Oct 31 13:36:21.780575 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.780489    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82dade06584e27c60fa6f915cd4a559fa6eff51d147214467bf04661c46f04f0"} err="failed to get container status \"82dade06584e27c60fa6f915cd4a559fa6eff51d147214467bf04661c46f04f0\": rpc error: code = NotFound desc = could not find container \"82dade06584e27c60fa6f915cd4a559fa6eff51d147214467bf04661c46f04f0\": container with ID starting with 82dade06584e27c60fa6f915cd4a559fa6eff51d147214467bf04661c46f04f0 not found: ID does not exist"
Oct 31 13:36:21.832382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.829935    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.832382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.829970    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a40e94-c560-4f09-b721-533261b4ff25-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.832382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.829985    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqmgw\" (UniqueName: \"kubernetes.io/projected/c1a40e94-c560-4f09-b721-533261b4ff25-kube-api-access-vqmgw\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.832382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:21.829998    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a40e94-c560-4f09-b721-533261b4ff25-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:22.139747 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:22.132727    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"]
Oct 31 13:36:22.151062 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:22.151024    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-79f4fb0f/e2e-lifecycle-model-79f4fb0f-kserve-7f8d784789-8fzgq"]
Oct 31 13:36:22.941253 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:22.941212    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1a40e94-c560-4f09-b721-533261b4ff25" path="/var/lib/kubelet/pods/c1a40e94-c560-4f09-b721-533261b4ff25/volumes"
Oct 31 13:36:23.653393 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:23.652817    4660 patch_prober.go:29] interesting pod/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:36:23.653393 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:23.652887    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" podUID="d86a5000-770e-409d-b23a-f836beb99b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:36:25.677003 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:25.676960    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"]
Oct 31 13:36:25.677632 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:25.677195    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq" podUID="f1fe6bcb-7c06-423e-88d6-0efe11f984b0" containerName="maas-api" containerID="cri-o://1784ba48f6ba5f22e7a1604f3605a8e307eb4089fe890ced19eb78366261169f" gracePeriod=30
Oct 31 13:36:26.235378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.218227    4660 generic.go:356] "Generic (PLEG): container finished" podID="f1fe6bcb-7c06-423e-88d6-0efe11f984b0" containerID="1784ba48f6ba5f22e7a1604f3605a8e307eb4089fe890ced19eb78366261169f" exitCode=0
Oct 31 13:36:26.235378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.218352    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq" event={"ID":"f1fe6bcb-7c06-423e-88d6-0efe11f984b0","Type":"ContainerDied","Data":"1784ba48f6ba5f22e7a1604f3605a8e307eb4089fe890ced19eb78366261169f"}
Oct 31 13:36:26.369316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.358588    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"]
Oct 31 13:36:26.369316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.359815    4660 cpu_manager.go:414] "containerMap: removing container" podUID="c1a40e94-c560-4f09-b721-533261b4ff25" containerName="main"
Oct 31 13:36:26.369316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.359839    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a40e94-c560-4f09-b721-533261b4ff25" containerName="main"
Oct 31 13:36:26.369316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.360099    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1a40e94-c560-4f09-b721-533261b4ff25" containerName="main"
Oct 31 13:36:26.456624 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.456578    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"]
Oct 31 13:36:26.456828 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.456719    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.487005 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.486916    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-15dc2f-openshift-default-dockercfg-5tkcp\""
Oct 31 13:36:26.513365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.513330    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.525981 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.525943    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.526395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.526367    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.535022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.534989    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbxcg\" (UniqueName: \"kubernetes.io/projected/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-kube-api-access-cbxcg\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.535361 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.535340    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.535579 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.535552    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.535754 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.535731    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.535931 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.535916    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.536158 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.536145    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.647165 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.646570    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.651954 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.651571    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.651954 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.651626    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.651954 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.651731    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.651954 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.651842    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.651954 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.651895    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbxcg\" (UniqueName: \"kubernetes.io/projected/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-kube-api-access-cbxcg\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.652275 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.652021    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.652275 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.652087    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.652275 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.652108    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.653887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.652596    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.653887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.653661    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.653887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.653854    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.655332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.655271    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:26.655470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:26.655455    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:27.230064 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:27.229977    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:27.230064 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:27.230052    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbxcg\" (UniqueName: \"kubernetes.io/projected/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-kube-api-access-cbxcg\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:27.234589 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:27.234553    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:27.239613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:27.239583    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:27.461484 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:27.461449    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:28.147751 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.147710    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:36:28.244873 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.244788    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"
Oct 31 13:36:28.245374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.244797    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq" event={"ID":"f1fe6bcb-7c06-423e-88d6-0efe11f984b0","Type":"ContainerDied","Data":"6e14057e8a9c1ac8cecde59ca36febc5f08e2ca490fa3ca53719d051b95b1c7a"}
Oct 31 13:36:28.245374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.244935    4660 scope.go:122] "RemoveContainer" containerID="1784ba48f6ba5f22e7a1604f3605a8e307eb4089fe890ced19eb78366261169f"
Oct 31 13:36:28.286966 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.284885    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-maas-api-tls\") pod \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\" (UID: \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\") "
Oct 31 13:36:28.286966 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.284993    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-metrics-certs\") pod \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\" (UID: \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\") "
Oct 31 13:36:28.286966 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.285108    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-openshift-service-ca\") pod \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\" (UID: \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\") "
Oct 31 13:36:28.286966 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.285375    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-kube-api-access-qx8sh\" (UniqueName: \"kubernetes.io/projected/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-kube-api-access-qx8sh\") pod \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\" (UID: \"f1fe6bcb-7c06-423e-88d6-0efe11f984b0\") "
Oct 31 13:36:28.288675 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.288641    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-openshift-service-ca" pod "f1fe6bcb-7c06-423e-88d6-0efe11f984b0" (UID: "f1fe6bcb-7c06-423e-88d6-0efe11f984b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:28.291639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.291521    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-maas-api-tls" pod "f1fe6bcb-7c06-423e-88d6-0efe11f984b0" (UID: "f1fe6bcb-7c06-423e-88d6-0efe11f984b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:36:28.294775 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.294752    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-metrics-certs" pod "f1fe6bcb-7c06-423e-88d6-0efe11f984b0" (UID: "f1fe6bcb-7c06-423e-88d6-0efe11f984b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:36:28.294920 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.294812    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-kube-api-access-qx8sh" pod "f1fe6bcb-7c06-423e-88d6-0efe11f984b0" (UID: "f1fe6bcb-7c06-423e-88d6-0efe11f984b0"). InnerVolumeSpecName "kube-api-access-qx8sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:28.472318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.467074    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:28.472318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.467112    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:28.472318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.467128    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:28.472318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.467146    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx8sh\" (UniqueName: \"kubernetes.io/projected/f1fe6bcb-7c06-423e-88d6-0efe11f984b0-kube-api-access-qx8sh\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:28.736339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.734980    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"]
Oct 31 13:36:28.778843 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:28.778723    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-752869-6f4447749-rthlq"]
Oct 31 13:36:29.937804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:29.937701    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"]
Oct 31 13:36:29.938280 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:36:29.937840    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc4155a_edd2_4907_a3c1_741f6ecefcdb.slice/crio-aeeee131c43d374c4009ebdde5c39022041a7fb3ad7e382ec4180d9ac7f199ae WatchSource:0}: Error finding container aeeee131c43d374c4009ebdde5c39022041a7fb3ad7e382ec4180d9ac7f199ae: Status 404 returned error can't find the container with id aeeee131c43d374c4009ebdde5c39022041a7fb3ad7e382ec4180d9ac7f199ae
Oct 31 13:36:29.959065 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:29.959020    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:29.959251 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:29.959105    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:29.959251 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:29.959145    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:30.268580 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:30.268490    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g" event={"ID":"1fc4155a-edd2-4907-a3c1-741f6ecefcdb","Type":"ContainerStarted","Data":"aeeee131c43d374c4009ebdde5c39022041a7fb3ad7e382ec4180d9ac7f199ae"}
Oct 31 13:36:30.539479 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:30.539403    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1fe6bcb-7c06-423e-88d6-0efe11f984b0" path="/var/lib/kubelet/pods/f1fe6bcb-7c06-423e-88d6-0efe11f984b0/volumes"
Oct 31 13:36:31.343596 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:31.342091    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g" event={"ID":"1fc4155a-edd2-4907-a3c1-741f6ecefcdb","Type":"ContainerStarted","Data":"b79a596ebcc20a2628bd7ee4c7474d1c523b98daefbbfbe6939d199ab808582f"}
Oct 31 13:36:31.470375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:31.415790    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g" podStartSLOduration=5.415768695 podStartE2EDuration="5.415768695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:31.40380655 +0000 UTC m=+1991.022465714" watchObservedRunningTime="2026-10-31 13:36:31.415768695 +0000 UTC m=+1991.034427859"
Oct 31 13:36:31.482116 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:31.474236    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 31 13:36:31.482116 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:31.474316    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g" podUID="1fc4155a-edd2-4907-a3c1-741f6ecefcdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 31 13:36:31.486353 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:31.471214    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:32.462244 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:32.462209    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 31 13:36:32.462910 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:32.462263    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g" podUID="1fc4155a-edd2-4907-a3c1-741f6ecefcdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 31 13:36:33.462468 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:33.462429    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 31 13:36:33.462964 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:33.462489    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g" podUID="1fc4155a-edd2-4907-a3c1-741f6ecefcdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 31 13:36:33.885524 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:33.885471    4660 patch_prober.go:29] interesting pod/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:36:33.885824 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:33.885567    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:36:34.477934 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:34.477748    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 31 13:36:34.477934 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:34.477820    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g" podUID="1fc4155a-edd2-4907-a3c1-741f6ecefcdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 31 13:36:35.552412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:35.463528    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 31 13:36:35.552412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:35.463587    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g" podUID="1fc4155a-edd2-4907-a3c1-741f6ecefcdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 31 13:36:36.462789 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:36.462706    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 31 13:36:36.462789 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:36.462776    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g" podUID="1fc4155a-edd2-4907-a3c1-741f6ecefcdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 31 13:36:37.261336 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.253578    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"]
Oct 31 13:36:37.261336 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.254734    4660 cpu_manager.go:414] "containerMap: removing container" podUID="f1fe6bcb-7c06-423e-88d6-0efe11f984b0" containerName="maas-api"
Oct 31 13:36:37.261336 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.254758    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1fe6bcb-7c06-423e-88d6-0efe11f984b0" containerName="maas-api"
Oct 31 13:36:37.282871 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.262209    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1fe6bcb-7c06-423e-88d6-0efe11f984b0" containerName="maas-api"
Oct 31 13:36:37.303386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.302972    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.307260 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.307224    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.307885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.307859    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.308186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.308167    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.308523 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.308504    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.310709 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.308683    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.310892 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.310871    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqv4t\" (UniqueName: \"kubernetes.io/projected/30ba0e0d-84ed-4caf-9351-a65cbf400dec-kube-api-access-wqv4t\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.311099 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.311081    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.316648 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.311311    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.322742 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.313214    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"]
Oct 31 13:36:37.322742 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.320636    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-79f4fb0f-openshift-default-dockercfg-shgv6\""
Oct 31 13:36:37.323067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.323044    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.471673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.449446    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.471673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.449615    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.471673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.449671    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.471673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.449740    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.471673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.449792    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqv4t\" (UniqueName: \"kubernetes.io/projected/30ba0e0d-84ed-4caf-9351-a65cbf400dec-kube-api-access-wqv4t\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.471673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.449876    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.471673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.449987    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.471673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.450101    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.471673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.450214    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.471673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.450521    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.471673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.450637    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.471673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.450883    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.471673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.451307    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.471673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.451838    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.497983 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.496684    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.527416 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.525058    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.528620 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.528514    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 31 13:36:37.528620 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.528566    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g" podUID="1fc4155a-edd2-4907-a3c1-741f6ecefcdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 31 13:36:37.534593 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.534530    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqv4t\" (UniqueName: \"kubernetes.io/projected/30ba0e0d-84ed-4caf-9351-a65cbf400dec-kube-api-access-wqv4t\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.540401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.540206    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:37.738596 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:37.738556    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:38.463434 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:38.462941    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 31 13:36:38.463434 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:38.463007    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g" podUID="1fc4155a-edd2-4907-a3c1-741f6ecefcdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 31 13:36:38.494625 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:38.491447    4660 patch_prober.go:29] interesting pod/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:36:38.494625 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:38.491525    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" podUID="d86a5000-770e-409d-b23a-f836beb99b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:36:38.626074 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:36:38.614409    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ba0e0d_84ed_4caf_9351_a65cbf400dec.slice/crio-9b2162dded7b4eab544ce892b89c0fa3d1cb4b2df622807924f5a4cc48736bec WatchSource:0}: Error finding container 9b2162dded7b4eab544ce892b89c0fa3d1cb4b2df622807924f5a4cc48736bec: Status 404 returned error can't find the container with id 9b2162dded7b4eab544ce892b89c0fa3d1cb4b2df622807924f5a4cc48736bec
Oct 31 13:36:38.626074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:38.618626    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:38.626074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:38.618712    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:38.626074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:38.618752    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:38.634285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:38.634211    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"]
Oct 31 13:36:38.722906 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:38.722866    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2" event={"ID":"30ba0e0d-84ed-4caf-9351-a65cbf400dec","Type":"ContainerStarted","Data":"9b2162dded7b4eab544ce892b89c0fa3d1cb4b2df622807924f5a4cc48736bec"}
Oct 31 13:36:39.469614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:39.469576    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:39.470222 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:39.470144    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:39.472179 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:39.472155    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:36:39.759260 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:39.759107    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2" event={"ID":"30ba0e0d-84ed-4caf-9351-a65cbf400dec","Type":"ContainerStarted","Data":"a1cde7da2e41874cbca071036490cb087a12c18535d3bd2efa0446f6a845b356"}
Oct 31 13:36:39.793385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:39.793319    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2" podStartSLOduration=2.7932690559999998 podStartE2EDuration="2.793269056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:39.788955853 +0000 UTC m=+1999.407615021" watchObservedRunningTime="2026-10-31 13:36:39.793269056 +0000 UTC m=+1999.411928221"
Oct 31 13:36:40.740139 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:40.740095    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:40.741217 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:40.741188    4660 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 13:36:40.741378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:40.741245    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2" podUID="30ba0e0d-84ed-4caf-9351-a65cbf400dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 13:36:41.921804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:41.742996    4660 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 13:36:41.921804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:41.743061    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2" podUID="30ba0e0d-84ed-4caf-9351-a65cbf400dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 13:36:42.740967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:42.740927    4660 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 13:36:42.741134 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:42.741003    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2" podUID="30ba0e0d-84ed-4caf-9351-a65cbf400dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 13:36:43.075267 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:43.075180    4660 generic.go:356] "Generic (PLEG): container finished" podID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerID="9d9a900557fc97ec5245a5cd81af8edb12e443771052545a7076b9191d5cffd8" exitCode=0
Oct 31 13:36:43.075267 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:43.075252    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" event={"ID":"6249baed-5e62-4178-a9a8-ffe554407c0b","Type":"ContainerDied","Data":"9d9a900557fc97ec5245a5cd81af8edb12e443771052545a7076b9191d5cffd8"}
Oct 31 13:36:43.075791 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:43.075772    4660 scope.go:122] "RemoveContainer" containerID="9d9a900557fc97ec5245a5cd81af8edb12e443771052545a7076b9191d5cffd8"
Oct 31 13:36:43.077257 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:43.077234    4660 generic.go:356] "Generic (PLEG): container finished" podID="d86a5000-770e-409d-b23a-f836beb99b08" containerID="26aaf65ec927aef941594f84586d17582de30dd442b3920221eb17f2babc4506" exitCode=0
Oct 31 13:36:43.077412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:43.077286    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" event={"ID":"d86a5000-770e-409d-b23a-f836beb99b08","Type":"ContainerDied","Data":"26aaf65ec927aef941594f84586d17582de30dd442b3920221eb17f2babc4506"}
Oct 31 13:36:43.077795 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:43.077778    4660 scope.go:122] "RemoveContainer" containerID="26aaf65ec927aef941594f84586d17582de30dd442b3920221eb17f2babc4506"
Oct 31 13:36:43.077965 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:43.077938    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:43.078033 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:43.078018    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:43.078076 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:43.078066    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:43.079472 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:43.079444    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:43.079578 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:43.079509    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:43.079578 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:43.079543    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:43.777031 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:43.776054    4660 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 13:36:43.777031 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:43.776114    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2" podUID="30ba0e0d-84ed-4caf-9351-a65cbf400dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 13:36:44.177321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:44.167900    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" event={"ID":"d86a5000-770e-409d-b23a-f836beb99b08","Type":"ContainerStarted","Data":"0726287ce8f68afb6feb05e5741cae036ad6009153a1972070c985101a72a0dc"}
Oct 31 13:36:44.177321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:44.174275    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" event={"ID":"6249baed-5e62-4178-a9a8-ffe554407c0b","Type":"ContainerStarted","Data":"ca260e3a115d4f91f21f6219133cf4ba08eda93caa93c8823009b7d107850219"}
Oct 31 13:36:44.752321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:44.737812    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:36:44.752545 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:44.752488    4660 patch_prober.go:29] interesting pod/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 13:36:44.752632 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:44.752546    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" podUID="d86a5000-770e-409d-b23a-f836beb99b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 13:36:44.777316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:44.775635    4660 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 13:36:44.777316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:44.775706    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2" podUID="30ba0e0d-84ed-4caf-9351-a65cbf400dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 13:36:44.889880 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:44.889845    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:36:44.891068 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:44.891027    4660 patch_prober.go:29] interesting pod/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 13:36:44.891201 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:44.891102    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 13:36:45.726320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:45.719771    4660 patch_prober.go:29] interesting pod/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 13:36:45.726320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:45.719858    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" podUID="d86a5000-770e-409d-b23a-f836beb99b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 13:36:45.756319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:45.755753    4660 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 13:36:45.756319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:45.755841    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2" podUID="30ba0e0d-84ed-4caf-9351-a65cbf400dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 13:36:45.891323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:45.890671    4660 patch_prober.go:29] interesting pod/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 13:36:45.891323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:45.890755    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 13:36:46.725324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:46.724699    4660 patch_prober.go:29] interesting pod/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 13:36:46.725324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:46.724777    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" podUID="d86a5000-770e-409d-b23a-f836beb99b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 13:36:46.779323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:46.777391    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:46.779323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:46.778830    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:46.780206 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:46.780185    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:36:46.898337 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:46.896837    4660 patch_prober.go:29] interesting pod/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 13:36:46.898337 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:46.896892    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 13:36:47.724421 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:47.724354    4660 patch_prober.go:29] interesting pod/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 13:36:47.724737 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:47.724706    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" podUID="d86a5000-770e-409d-b23a-f836beb99b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 13:36:47.851322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:47.850871    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:36:47.889533 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:47.889499    4660 patch_prober.go:29] interesting pod/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 13:36:47.889726 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:47.889565    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 13:36:48.724317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:48.720738    4660 patch_prober.go:29] interesting pod/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 13:36:48.724317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:48.720811    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" podUID="d86a5000-770e-409d-b23a-f836beb99b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 13:36:48.889347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:48.889316    4660 patch_prober.go:29] interesting pod/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 13:36:48.889815 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:48.889788    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 13:36:49.758486 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:49.758445    4660 patch_prober.go:29] interesting pod/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 13:36:49.759321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:49.758747    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" podUID="d86a5000-770e-409d-b23a-f836beb99b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 13:36:49.890351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:49.890285    4660 patch_prober.go:29] interesting pod/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 13:36:49.890855 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:49.890367    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 13:36:50.781528 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:50.781487    4660 patch_prober.go:29] interesting pod/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 13:36:50.781843 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:50.781806    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" podUID="d86a5000-770e-409d-b23a-f836beb99b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 13:36:50.896839 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:50.896798    4660 patch_prober.go:29] interesting pod/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 13:36:50.897492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:50.897455    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 13:36:51.072584 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:51.072489    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:36:51.180778 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:36:51.180739    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:36:51.741088 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:51.741053    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:36:51.742680 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:51.742656    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:36:51.743549 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:51.743532    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:36:51.890099 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:51.890057    4660 patch_prober.go:29] interesting pod/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 13:36:51.890276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:51.890134    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 13:36:52.051372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.050924    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"]
Oct 31 13:36:52.059717 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.059686    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.063065 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.063033    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"]
Oct 31 13:36:52.083057 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.082608    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.083279 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.083209    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"]
Oct 31 13:36:52.093254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.093219    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"]
Oct 31 13:36:52.196225 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.196189    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.196474 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.196261    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0956d488-da8e-4597-93e5-4a6918b1887d-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.196474 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.196414    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.196604 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.196530    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kbk4\" (UniqueName: \"kubernetes.io/projected/0956d488-da8e-4597-93e5-4a6918b1887d-kube-api-access-2kbk4\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.196604 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.196565    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.196604 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.196583    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.196777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.196666    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.196777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.196760    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.196893 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.196781    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.196953 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.196925    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0956d488-da8e-4597-93e5-4a6918b1887d-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.197014 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.196984    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.197064 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.197030    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.197159 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.197144    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0956d488-da8e-4597-93e5-4a6918b1887d-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.197215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.197173    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.197262 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.197209    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhg9q\" (UniqueName: \"kubernetes.io/projected/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-kube-api-access-nhg9q\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.197342 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.197266    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.197391 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.197338    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.197391 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.197359    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.299144 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299103    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kbk4\" (UniqueName: \"kubernetes.io/projected/0956d488-da8e-4597-93e5-4a6918b1887d-kube-api-access-2kbk4\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.299373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299153    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.299373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299177    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.299373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299206    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.299373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299255    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.299373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299278    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.299373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299335    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0956d488-da8e-4597-93e5-4a6918b1887d-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.299373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299362    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.299745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299400    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.299745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299482    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0956d488-da8e-4597-93e5-4a6918b1887d-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.299745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299513    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.299745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299552    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhg9q\" (UniqueName: \"kubernetes.io/projected/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-kube-api-access-nhg9q\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.299745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299599    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.299745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299632    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.299745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299662    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.300143 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299884    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.300143 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.299965    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0956d488-da8e-4597-93e5-4a6918b1887d-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.300143 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.300040    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.300626 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.300599    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.300740 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.300704    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.300816 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.300748    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.300977 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.300950    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.301172 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.301149    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.301737 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.301665    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0956d488-da8e-4597-93e5-4a6918b1887d-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.301737 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.301713    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.301992 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.301971    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.302348 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.302271    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.305141 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.305113    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.312238 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.311689    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.327314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.324386    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.332664 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.332599    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0956d488-da8e-4597-93e5-4a6918b1887d-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.333517 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.333488    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhg9q\" (UniqueName: \"kubernetes.io/projected/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-kube-api-access-nhg9q\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.334758 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.334711    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.336080 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.336033    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0956d488-da8e-4597-93e5-4a6918b1887d-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.336627 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.336609    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.339147 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.339105    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kbk4\" (UniqueName: \"kubernetes.io/projected/0956d488-da8e-4597-93e5-4a6918b1887d-kube-api-access-2kbk4\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.417555 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.417516    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:52.421554 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.421523    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:52.723721 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:36:52.723685    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48afff97_c9ef_4c8c_8304_9abb9ce5a0b2.slice/crio-d4a3e2d42f4a3d39b2c654117887acec0ad90141eafb5d2fcc064f02de465780 WatchSource:0}: Error finding container d4a3e2d42f4a3d39b2c654117887acec0ad90141eafb5d2fcc064f02de465780: Status 404 returned error can't find the container with id d4a3e2d42f4a3d39b2c654117887acec0ad90141eafb5d2fcc064f02de465780
Oct 31 13:36:52.725052 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.725026    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"]
Oct 31 13:36:52.727189 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.727143    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:52.727308 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.727237    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:52.727308 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.727282    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:52.742599 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:36:52.742562    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0956d488_da8e_4597_93e5_4a6918b1887d.slice/crio-26a437845844a1004767aa7c9dd58cad51103a6b398d596c5fd774ebc33de815 WatchSource:0}: Error finding container 26a437845844a1004767aa7c9dd58cad51103a6b398d596c5fd774ebc33de815: Status 404 returned error can't find the container with id 26a437845844a1004767aa7c9dd58cad51103a6b398d596c5fd774ebc33de815
Oct 31 13:36:52.745021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.744983    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"]
Oct 31 13:36:52.746247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.746207    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:52.746374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.746309    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:52.746374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.746355    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:36:52.900600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.899093    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:36:52.905347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:52.902139    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:36:53.137196 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:53.137138    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569" event={"ID":"0956d488-da8e-4597-93e5-4a6918b1887d","Type":"ContainerStarted","Data":"3d69024c279ef4f7fd6f76411c850c87abcba3d7e08b2bab2f7b10387d6491dd"}
Oct 31 13:36:53.137818 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:53.137209    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569" event={"ID":"0956d488-da8e-4597-93e5-4a6918b1887d","Type":"ContainerStarted","Data":"26a437845844a1004767aa7c9dd58cad51103a6b398d596c5fd774ebc33de815"}
Oct 31 13:36:53.140463 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:53.140435    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m" event={"ID":"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2","Type":"ContainerStarted","Data":"77978a4994824c4277220fda50e442ad390339f513b3edcfe0227f54fa4b8853"}
Oct 31 13:36:53.140745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:53.140726    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m" event={"ID":"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2","Type":"ContainerStarted","Data":"d4a3e2d42f4a3d39b2c654117887acec0ad90141eafb5d2fcc064f02de465780"}
Oct 31 13:36:53.165098 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:53.165030    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569" podStartSLOduration=1.165009805 podStartE2EDuration="1.165009805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:53.163149429 +0000 UTC m=+2012.781808594" watchObservedRunningTime="2026-10-31 13:36:53.165009805 +0000 UTC m=+2012.783668970"
Oct 31 13:36:53.230622 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:53.230562    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m" podStartSLOduration=1.230545219 podStartE2EDuration="1.230545219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:53.188890684 +0000 UTC m=+2012.807549848" watchObservedRunningTime="2026-10-31 13:36:53.230545219 +0000 UTC m=+2012.849204380"
Oct 31 13:36:53.421398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:53.417857    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:36:53.421398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:53.419450    4660 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 13:36:53.421398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:53.419491    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m" podUID="48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 13:36:53.421949 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:53.421927    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:36:53.423919 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:53.423883    4660 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 13:36:53.424050 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:53.423926    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569" podUID="0956d488-da8e-4597-93e5-4a6918b1887d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 13:36:54.222318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:54.221399    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"]
Oct 31 13:36:54.222318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:54.221656    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2" podUID="30ba0e0d-84ed-4caf-9351-a65cbf400dec" containerName="istio-proxy" containerID="cri-o://a1cde7da2e41874cbca071036490cb087a12c18535d3bd2efa0446f6a845b356" gracePeriod=30
Oct 31 13:36:54.537376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:54.493998    4660 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569" secret="" err="secret \"e2e-mt-79f4fb0f-openshift-default-dockercfg-shgv6\" not found"
Oct 31 13:36:54.575366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:54.538471    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"]
Oct 31 13:36:54.575366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:54.571465    4660 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 13:36:54.575366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:54.571525    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m" podUID="48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 13:36:54.577359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:54.577326    4660 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m" secret="" err="secret \"e2e-mt-79f4fb0f-openshift-default-dockercfg-shgv6\" not found"
Oct 31 13:36:54.593356 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:54.582483    4660 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 13:36:54.593356 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:54.582552    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569" podUID="0956d488-da8e-4597-93e5-4a6918b1887d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 13:36:54.728616 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:54.728579    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"]
Oct 31 13:36:55.422151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:55.421355    4660 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 13:36:55.422151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:55.421430    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m" podUID="48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 13:36:55.422151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:55.422063    4660 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 13:36:55.422151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:55.422098    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569" podUID="0956d488-da8e-4597-93e5-4a6918b1887d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 13:36:55.469498 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:55.468285    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m" podUID="48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" containerName="istio-proxy" containerID="cri-o://77978a4994824c4277220fda50e442ad390339f513b3edcfe0227f54fa4b8853" gracePeriod=30
Oct 31 13:36:55.469498 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:55.468920    4660 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569" secret="" err="secret \"e2e-mt-79f4fb0f-openshift-default-dockercfg-shgv6\" not found"
Oct 31 13:36:56.426318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:56.425724    4660 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 13:36:56.426318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:56.425805    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569" podUID="0956d488-da8e-4597-93e5-4a6918b1887d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 13:36:56.477395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:56.477356    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569" podUID="0956d488-da8e-4597-93e5-4a6918b1887d" containerName="istio-proxy" containerID="cri-o://3d69024c279ef4f7fd6f76411c850c87abcba3d7e08b2bab2f7b10387d6491dd" gracePeriod=30
Oct 31 13:36:59.842316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:59.841966    4660 generic.go:356] "Generic (PLEG): container finished" podID="30ba0e0d-84ed-4caf-9351-a65cbf400dec" containerID="a1cde7da2e41874cbca071036490cb087a12c18535d3bd2efa0446f6a845b356" exitCode=0
Oct 31 13:36:59.842316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:36:59.842283    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2" event={"ID":"30ba0e0d-84ed-4caf-9351-a65cbf400dec","Type":"ContainerDied","Data":"a1cde7da2e41874cbca071036490cb087a12c18535d3bd2efa0446f6a845b356"}
Oct 31 13:37:00.475272 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:00.459877    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"]
Oct 31 13:37:00.475272 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:00.461406    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" containerID="cri-o://3a3aa6fd747210a7a1571b2ee7ddf5c29e9d576c9ddea2ff14a5fd0e0ce22ab1" gracePeriod=30
Oct 31 13:37:00.769991 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:00.768411    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:00.826319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:00.824359    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:00.826319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:00.825737    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:37:00.993073 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:00.984865    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"]
Oct 31 13:37:00.993073 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:00.985188    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj" podUID="11d0989b-0ce8-4041-bf02-e9af6b10c3ae" containerName="istio-proxy" containerID="cri-o://117a3b16d6b5851cd1cae61ecf0c84b5e0f2f7acc73a32f5b93946d4c0c0096a" gracePeriod=30
Oct 31 13:37:01.022595 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.022493    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:01.241473 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.241438    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:37:01.348334 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.342282    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-envoy\") pod \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") "
Oct 31 13:37:01.366831 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.366774    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-workload-certs\") pod \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") "
Oct 31 13:37:01.384855 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.371632    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-workload-certs" pod "30ba0e0d-84ed-4caf-9351-a65cbf400dec" (UID: "30ba0e0d-84ed-4caf-9351-a65cbf400dec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:01.389014 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.386720    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-envoy" pod "30ba0e0d-84ed-4caf-9351-a65cbf400dec" (UID: "30ba0e0d-84ed-4caf-9351-a65cbf400dec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:01.391409 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.389234    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-podinfo" pod "30ba0e0d-84ed-4caf-9351-a65cbf400dec" (UID: "30ba0e0d-84ed-4caf-9351-a65cbf400dec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:37:01.418849 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.374462    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-podinfo\") pod \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") "
Oct 31 13:37:01.419411 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.419389    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istiod-ca-cert\") pod \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") "
Oct 31 13:37:01.419832 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.419803    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istiod-ca-cert" pod "30ba0e0d-84ed-4caf-9351-a65cbf400dec" (UID: "30ba0e0d-84ed-4caf-9351-a65cbf400dec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:01.432459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.424948    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-token\" (UniqueName: \"kubernetes.io/projected/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-token\") pod \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") "
Oct 31 13:37:01.432459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.425432    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-credential-socket\") pod \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") "
Oct 31 13:37:01.432459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.426649    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-credential-socket" pod "30ba0e0d-84ed-4caf-9351-a65cbf400dec" (UID: "30ba0e0d-84ed-4caf-9351-a65cbf400dec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:01.432459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.426777    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-workload-socket\") pod \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") "
Oct 31 13:37:01.432459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.427013    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-workload-socket" pod "30ba0e0d-84ed-4caf-9351-a65cbf400dec" (UID: "30ba0e0d-84ed-4caf-9351-a65cbf400dec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:01.432459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.427231    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-data\") pod \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") "
Oct 31 13:37:01.432459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.427478    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-data" pod "30ba0e0d-84ed-4caf-9351-a65cbf400dec" (UID: "30ba0e0d-84ed-4caf-9351-a65cbf400dec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:01.432459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.427608    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ba0e0d-84ed-4caf-9351-a65cbf400dec-kube-api-access-wqv4t\" (UniqueName: \"kubernetes.io/projected/30ba0e0d-84ed-4caf-9351-a65cbf400dec-kube-api-access-wqv4t\") pod \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\" (UID: \"30ba0e0d-84ed-4caf-9351-a65cbf400dec\") "
Oct 31 13:37:01.432459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.428815    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:01.432459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.428842    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:01.432459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.428857    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:01.432459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.428872    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:01.432459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.428886    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ba0e0d-84ed-4caf-9351-a65cbf400dec-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:01.432459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.428902    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:01.432459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.428917    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:01.433512 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.433478    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ba0e0d-84ed-4caf-9351-a65cbf400dec-kube-api-access-wqv4t" pod "30ba0e0d-84ed-4caf-9351-a65cbf400dec" (UID: "30ba0e0d-84ed-4caf-9351-a65cbf400dec"). InnerVolumeSpecName "kube-api-access-wqv4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:01.479532 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.479493    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-token" pod "30ba0e0d-84ed-4caf-9351-a65cbf400dec" (UID: "30ba0e0d-84ed-4caf-9351-a65cbf400dec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:01.531361 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.531249    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ba0e0d-84ed-4caf-9351-a65cbf400dec-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:01.531361 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:01.531313    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqv4t\" (UniqueName: \"kubernetes.io/projected/30ba0e0d-84ed-4caf-9351-a65cbf400dec-kube-api-access-wqv4t\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:02.140202 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:02.140164    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2" event={"ID":"30ba0e0d-84ed-4caf-9351-a65cbf400dec","Type":"ContainerDied","Data":"9b2162dded7b4eab544ce892b89c0fa3d1cb4b2df622807924f5a4cc48736bec"}
Oct 31 13:37:02.140947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:02.140909    4660 scope.go:122] "RemoveContainer" containerID="a1cde7da2e41874cbca071036490cb087a12c18535d3bd2efa0446f6a845b356"
Oct 31 13:37:02.141324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:02.141281    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"
Oct 31 13:37:02.300669 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:02.300633    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"]
Oct 31 13:37:02.344319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:02.344263    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:37:02.344519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:02.344345    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:37:02.347364 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:02.347331    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-7cxj2"]
Oct 31 13:37:02.532375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:02.532232    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30ba0e0d-84ed-4caf-9351-a65cbf400dec" path="/var/lib/kubelet/pods/30ba0e0d-84ed-4caf-9351-a65cbf400dec/volumes"
Oct 31 13:37:05.299612 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:05.299570    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:05.580958 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:05.573393    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:05.631760 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:05.628490    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:37:05.697238 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:05.696210    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:05.806030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:05.805820    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:37:05.818153 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:05.817391    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:06.240514 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:06.237947    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:06.317872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:06.317736    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:06.633376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:06.630426    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:06.696920 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:06.696749    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:06.732125 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:06.732092    4660 generic.go:356] "Generic (PLEG): container finished" podID="11d0989b-0ce8-4041-bf02-e9af6b10c3ae" containerID="117a3b16d6b5851cd1cae61ecf0c84b5e0f2f7acc73a32f5b93946d4c0c0096a" exitCode=0
Oct 31 13:37:06.735240 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:06.735214    4660 generic.go:356] "Generic (PLEG): container finished" podID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerID="3a3aa6fd747210a7a1571b2ee7ddf5c29e9d576c9ddea2ff14a5fd0e0ce22ab1" exitCode=0
Oct 31 13:37:06.809417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:06.809371    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:06.856912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:06.856869    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:06.908519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:06.907137    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj" event={"ID":"11d0989b-0ce8-4041-bf02-e9af6b10c3ae","Type":"ContainerDied","Data":"117a3b16d6b5851cd1cae61ecf0c84b5e0f2f7acc73a32f5b93946d4c0c0096a"}
Oct 31 13:37:06.908519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:06.907215    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" event={"ID":"63fba834-3ec6-4542-97b5-5ddac5d4d450","Type":"ContainerDied","Data":"3a3aa6fd747210a7a1571b2ee7ddf5c29e9d576c9ddea2ff14a5fd0e0ce22ab1"}
Oct 31 13:37:06.908519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:06.907285    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:37:06.908519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:06.907354    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:37:06.908519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:06.907388    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:37:06.908519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:06.907423    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:37:06.913349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:06.913322    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:37:07.080340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:07.076836    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:37:07.080340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:07.076903    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 13:37:07.323774 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:07.321433    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:37:07.341872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:07.336374    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:37:07.441720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:07.441678    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:37:07.497315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:07.492965    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:37:07.497315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:07.493018    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:37:07.735746 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:07.733838    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:37:07.785660 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:07.785482    4660 generic.go:356] "Generic (PLEG): container finished" podID="0956d488-da8e-4597-93e5-4a6918b1887d" containerID="3d69024c279ef4f7fd6f76411c850c87abcba3d7e08b2bab2f7b10387d6491dd" exitCode=0
Oct 31 13:37:07.785660 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:07.785589    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569" event={"ID":"0956d488-da8e-4597-93e5-4a6918b1887d","Type":"ContainerDied","Data":"3d69024c279ef4f7fd6f76411c850c87abcba3d7e08b2bab2f7b10387d6491dd"}
Oct 31 13:37:07.791276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:07.788744    4660 generic.go:356] "Generic (PLEG): container finished" podID="48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" containerID="77978a4994824c4277220fda50e442ad390339f513b3edcfe0227f54fa4b8853" exitCode=0
Oct 31 13:37:07.791276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:07.789949    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m" event={"ID":"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2","Type":"ContainerDied","Data":"77978a4994824c4277220fda50e442ad390339f513b3edcfe0227f54fa4b8853"}
Oct 31 13:37:07.939225 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:07.919173    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:37:07.939225 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:07.931883    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.043844    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-envoy\") pod \"63fba834-3ec6-4542-97b5-5ddac5d4d450\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.043950    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-token\" (UniqueName: \"kubernetes.io/projected/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-token\") pod \"63fba834-3ec6-4542-97b5-5ddac5d4d450\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.043984    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-data\") pod \"63fba834-3ec6-4542-97b5-5ddac5d4d450\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.044031    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-workload-socket\") pod \"63fba834-3ec6-4542-97b5-5ddac5d4d450\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.044088    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-podinfo\") pod \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.044133    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63fba834-3ec6-4542-97b5-5ddac5d4d450-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fba834-3ec6-4542-97b5-5ddac5d4d450-istiod-ca-cert\") pod \"63fba834-3ec6-4542-97b5-5ddac5d4d450\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.044199    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-podinfo\") pod \"63fba834-3ec6-4542-97b5-5ddac5d4d450\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.044232    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istiod-ca-cert\") pod \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.044259    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-workload-certs\") pod \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.044373    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63fba834-3ec6-4542-97b5-5ddac5d4d450-kube-api-access-4zmbb\" (UniqueName: \"kubernetes.io/projected/63fba834-3ec6-4542-97b5-5ddac5d4d450-kube-api-access-4zmbb\") pod \"63fba834-3ec6-4542-97b5-5ddac5d4d450\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.044415    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-token\" (UniqueName: \"kubernetes.io/projected/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-token\") pod \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.044481    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-kube-api-access-5hggg\" (UniqueName: \"kubernetes.io/projected/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-kube-api-access-5hggg\") pod \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.044526    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-credential-socket\") pod \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.044603    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-data\") pod \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.044760    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-workload-certs\") pod \"63fba834-3ec6-4542-97b5-5ddac5d4d450\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.044820    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-workload-socket\") pod \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.044864    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-credential-socket\") pod \"63fba834-3ec6-4542-97b5-5ddac5d4d450\" (UID: \"63fba834-3ec6-4542-97b5-5ddac5d4d450\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.044911    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-envoy\") pod \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\" (UID: \"11d0989b-0ce8-4041-bf02-e9af6b10c3ae\") "
Oct 31 13:37:08.054030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.050507    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-workload-certs" pod "11d0989b-0ce8-4041-bf02-e9af6b10c3ae" (UID: "11d0989b-0ce8-4041-bf02-e9af6b10c3ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.056886 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.056857    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.058980 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.058944    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-data" pod "11d0989b-0ce8-4041-bf02-e9af6b10c3ae" (UID: "11d0989b-0ce8-4041-bf02-e9af6b10c3ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.060495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.060334    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63fba834-3ec6-4542-97b5-5ddac5d4d450-kube-api-access-4zmbb" pod "63fba834-3ec6-4542-97b5-5ddac5d4d450" (UID: "63fba834-3ec6-4542-97b5-5ddac5d4d450"). InnerVolumeSpecName "kube-api-access-4zmbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:08.060495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.060480    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-workload-socket" pod "11d0989b-0ce8-4041-bf02-e9af6b10c3ae" (UID: "11d0989b-0ce8-4041-bf02-e9af6b10c3ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.061114 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.060947    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-credential-socket" pod "63fba834-3ec6-4542-97b5-5ddac5d4d450" (UID: "63fba834-3ec6-4542-97b5-5ddac5d4d450"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.061539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.061251    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-data" pod "63fba834-3ec6-4542-97b5-5ddac5d4d450" (UID: "63fba834-3ec6-4542-97b5-5ddac5d4d450"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.061539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.061372    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-workload-certs" pod "63fba834-3ec6-4542-97b5-5ddac5d4d450" (UID: "63fba834-3ec6-4542-97b5-5ddac5d4d450"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.074244 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.074045    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-credential-socket" pod "11d0989b-0ce8-4041-bf02-e9af6b10c3ae" (UID: "11d0989b-0ce8-4041-bf02-e9af6b10c3ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.077344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.074714    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istiod-ca-cert" pod "11d0989b-0ce8-4041-bf02-e9af6b10c3ae" (UID: "11d0989b-0ce8-4041-bf02-e9af6b10c3ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:08.079453 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.078107    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-kube-api-access-5hggg" pod "11d0989b-0ce8-4041-bf02-e9af6b10c3ae" (UID: "11d0989b-0ce8-4041-bf02-e9af6b10c3ae"). InnerVolumeSpecName "kube-api-access-5hggg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:08.079453 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.078738    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-token" pod "63fba834-3ec6-4542-97b5-5ddac5d4d450" (UID: "63fba834-3ec6-4542-97b5-5ddac5d4d450"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:08.079453 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.078830    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-token" pod "11d0989b-0ce8-4041-bf02-e9af6b10c3ae" (UID: "11d0989b-0ce8-4041-bf02-e9af6b10c3ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:08.079453 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.078908    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-envoy" pod "63fba834-3ec6-4542-97b5-5ddac5d4d450" (UID: "63fba834-3ec6-4542-97b5-5ddac5d4d450"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.083194 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.080427    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63fba834-3ec6-4542-97b5-5ddac5d4d450-istiod-ca-cert" pod "63fba834-3ec6-4542-97b5-5ddac5d4d450" (UID: "63fba834-3ec6-4542-97b5-5ddac5d4d450"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:08.083194 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.080914    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-podinfo" pod "11d0989b-0ce8-4041-bf02-e9af6b10c3ae" (UID: "11d0989b-0ce8-4041-bf02-e9af6b10c3ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:37:08.083194 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.080972    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-workload-socket" pod "63fba834-3ec6-4542-97b5-5ddac5d4d450" (UID: "63fba834-3ec6-4542-97b5-5ddac5d4d450"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.085456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.084924    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-podinfo" pod "63fba834-3ec6-4542-97b5-5ddac5d4d450" (UID: "63fba834-3ec6-4542-97b5-5ddac5d4d450"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:37:08.085790 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.085667    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-envoy" pod "11d0989b-0ce8-4041-bf02-e9af6b10c3ae" (UID: "11d0989b-0ce8-4041-bf02-e9af6b10c3ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.090673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.090641    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:37:08.162971 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.161094    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-istio-envoy\") pod \"0956d488-da8e-4597-93e5-4a6918b1887d\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") "
Oct 31 13:37:08.162971 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.161389    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0956d488-da8e-4597-93e5-4a6918b1887d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0956d488-da8e-4597-93e5-4a6918b1887d-istio-podinfo\") pod \"0956d488-da8e-4597-93e5-4a6918b1887d\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") "
Oct 31 13:37:08.163941 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.163312    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-istio-data\") pod \"0956d488-da8e-4597-93e5-4a6918b1887d\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") "
Oct 31 13:37:08.163941 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.163396    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-workload-socket\") pod \"0956d488-da8e-4597-93e5-4a6918b1887d\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") "
Oct 31 13:37:08.163941 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.163428    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0956d488-da8e-4597-93e5-4a6918b1887d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0956d488-da8e-4597-93e5-4a6918b1887d-istiod-ca-cert\") pod \"0956d488-da8e-4597-93e5-4a6918b1887d\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") "
Oct 31 13:37:08.163941 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.163460    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0956d488-da8e-4597-93e5-4a6918b1887d-kube-api-access-2kbk4\" (UniqueName: \"kubernetes.io/projected/0956d488-da8e-4597-93e5-4a6918b1887d-kube-api-access-2kbk4\") pod \"0956d488-da8e-4597-93e5-4a6918b1887d\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") "
Oct 31 13:37:08.163941 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.163549    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0956d488-da8e-4597-93e5-4a6918b1887d-istio-token\" (UniqueName: \"kubernetes.io/projected/0956d488-da8e-4597-93e5-4a6918b1887d-istio-token\") pod \"0956d488-da8e-4597-93e5-4a6918b1887d\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") "
Oct 31 13:37:08.163941 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.163716    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-workload-certs\") pod \"0956d488-da8e-4597-93e5-4a6918b1887d\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") "
Oct 31 13:37:08.163941 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.163753    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-credential-socket\") pod \"0956d488-da8e-4597-93e5-4a6918b1887d\" (UID: \"0956d488-da8e-4597-93e5-4a6918b1887d\") "
Oct 31 13:37:08.167564 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.167149    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0956d488-da8e-4597-93e5-4a6918b1887d-istiod-ca-cert" pod "0956d488-da8e-4597-93e5-4a6918b1887d" (UID: "0956d488-da8e-4597-93e5-4a6918b1887d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:08.168963 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.168859    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zmbb\" (UniqueName: \"kubernetes.io/projected/63fba834-3ec6-4542-97b5-5ddac5d4d450-kube-api-access-4zmbb\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.169121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169100    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.169537 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169273    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hggg\" (UniqueName: \"kubernetes.io/projected/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-kube-api-access-5hggg\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.170067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169656    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.170067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169685    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.170067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169711    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.170067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169727    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.170067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169741    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.170067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169754    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.170067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169769    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.170067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169782    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.170067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169795    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.170067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169808    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fba834-3ec6-4542-97b5-5ddac5d4d450-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.170067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169822    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0956d488-da8e-4597-93e5-4a6918b1887d-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.170067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169835    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.170067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169849    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fba834-3ec6-4542-97b5-5ddac5d4d450-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.170067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169864    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fba834-3ec6-4542-97b5-5ddac5d4d450-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.170067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.169877    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d0989b-0ce8-4041-bf02-e9af6b10c3ae-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.170896 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.170673    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-istio-data" pod "0956d488-da8e-4597-93e5-4a6918b1887d" (UID: "0956d488-da8e-4597-93e5-4a6918b1887d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.170956 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.170901    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-workload-socket" pod "0956d488-da8e-4597-93e5-4a6918b1887d" (UID: "0956d488-da8e-4597-93e5-4a6918b1887d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.171329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.171073    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-credential-socket" pod "0956d488-da8e-4597-93e5-4a6918b1887d" (UID: "0956d488-da8e-4597-93e5-4a6918b1887d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.171819 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.171521    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-workload-certs" pod "0956d488-da8e-4597-93e5-4a6918b1887d" (UID: "0956d488-da8e-4597-93e5-4a6918b1887d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.174929 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.174884    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0956d488-da8e-4597-93e5-4a6918b1887d-istio-token" pod "0956d488-da8e-4597-93e5-4a6918b1887d" (UID: "0956d488-da8e-4597-93e5-4a6918b1887d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:08.175191 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.175157    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0956d488-da8e-4597-93e5-4a6918b1887d-kube-api-access-2kbk4" pod "0956d488-da8e-4597-93e5-4a6918b1887d" (UID: "0956d488-da8e-4597-93e5-4a6918b1887d"). InnerVolumeSpecName "kube-api-access-2kbk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:08.175948 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.175912    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0956d488-da8e-4597-93e5-4a6918b1887d-istio-podinfo" pod "0956d488-da8e-4597-93e5-4a6918b1887d" (UID: "0956d488-da8e-4597-93e5-4a6918b1887d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:37:08.184606 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.184174    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-istio-envoy" pod "0956d488-da8e-4597-93e5-4a6918b1887d" (UID: "0956d488-da8e-4597-93e5-4a6918b1887d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.272110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.272083    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.272110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.272109    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.272378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.272121    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kbk4\" (UniqueName: \"kubernetes.io/projected/0956d488-da8e-4597-93e5-4a6918b1887d-kube-api-access-2kbk4\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.272378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.272131    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0956d488-da8e-4597-93e5-4a6918b1887d-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.272378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.272139    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.272378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.272149    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.272378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.272157    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0956d488-da8e-4597-93e5-4a6918b1887d-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.272378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.272165    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0956d488-da8e-4597-93e5-4a6918b1887d-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.277388 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.277364    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:37:08.373168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.373137    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-workload-certs\") pod \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") "
Oct 31 13:37:08.373723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.373208    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-envoy\") pod \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") "
Oct 31 13:37:08.373723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.373280    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istiod-ca-cert\") pod \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") "
Oct 31 13:37:08.373723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.373346    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-workload-socket\") pod \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") "
Oct 31 13:37:08.373723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.373447    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-workload-certs" pod "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" (UID: "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.373723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.373523    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-credential-socket\") pod \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") "
Oct 31 13:37:08.373723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.373609    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-kube-api-access-nhg9q\" (UniqueName: \"kubernetes.io/projected/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-kube-api-access-nhg9q\") pod \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") "
Oct 31 13:37:08.373723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.373634    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-workload-socket" pod "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" (UID: "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.373723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.373639    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-data\") pod \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") "
Oct 31 13:37:08.374363 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.373726    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-token\" (UniqueName: \"kubernetes.io/projected/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-token\") pod \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") "
Oct 31 13:37:08.374363 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.373848    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-credential-socket" pod "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" (UID: "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.374363 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.373864    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-data" pod "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" (UID: "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.374363 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.373895    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-podinfo\") pod \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\" (UID: \"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2\") "
Oct 31 13:37:08.374363 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.373761    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istiod-ca-cert" pod "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" (UID: "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:08.374786 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.374766    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.374858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.374789    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.374858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.374806    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.374858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.374821    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.374858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.374834    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.376109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.376080    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-podinfo" pod "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" (UID: "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:37:08.376224 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.376123    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-kube-api-access-nhg9q" pod "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" (UID: "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2"). InnerVolumeSpecName "kube-api-access-nhg9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:08.376807 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.376779    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-token" pod "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" (UID: "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:08.377457 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.377434    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-envoy" pod "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" (UID: "48afff97-c9ef-4c8c-8304-9abb9ce5a0b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:08.476018 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.475983    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.476018 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.476019    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.476018 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.476028    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhg9q\" (UniqueName: \"kubernetes.io/projected/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-kube-api-access-nhg9q\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.476307 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.476038    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:08.842758 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.842722    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"
Oct 31 13:37:08.845674 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.845624    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5" event={"ID":"63fba834-3ec6-4542-97b5-5ddac5d4d450","Type":"ContainerDied","Data":"b903f928d256c49d60ebaf5957934e258e110ec3b6736b6d483b413350d8e7a2"}
Oct 31 13:37:08.845890 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.845872    4660 scope.go:122] "RemoveContainer" containerID="3a3aa6fd747210a7a1571b2ee7ddf5c29e9d576c9ddea2ff14a5fd0e0ce22ab1"
Oct 31 13:37:08.847271 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.847249    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569" event={"ID":"0956d488-da8e-4597-93e5-4a6918b1887d","Type":"ContainerDied","Data":"26a437845844a1004767aa7c9dd58cad51103a6b398d596c5fd774ebc33de815"}
Oct 31 13:37:08.847589 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.847496    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"
Oct 31 13:37:08.863439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.863349    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"
Oct 31 13:37:08.864130 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.863723    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m" event={"ID":"48afff97-c9ef-4c8c-8304-9abb9ce5a0b2","Type":"ContainerDied","Data":"d4a3e2d42f4a3d39b2c654117887acec0ad90141eafb5d2fcc064f02de465780"}
Oct 31 13:37:08.871748 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.871227    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj" event={"ID":"11d0989b-0ce8-4041-bf02-e9af6b10c3ae","Type":"ContainerDied","Data":"a55bd516c828aa051c804c314029fe06d6f84c9c07def42d46f0db01a9ba35f9"}
Oct 31 13:37:08.871748 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.871399    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"
Oct 31 13:37:08.881700 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.881553    4660 scope.go:122] "RemoveContainer" containerID="3d69024c279ef4f7fd6f76411c850c87abcba3d7e08b2bab2f7b10387d6491dd"
Oct 31 13:37:08.896609 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.896588    4660 scope.go:122] "RemoveContainer" containerID="77978a4994824c4277220fda50e442ad390339f513b3edcfe0227f54fa4b8853"
Oct 31 13:37:08.944106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.944072    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"]
Oct 31 13:37:08.949203 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.949180    4660 scope.go:122] "RemoveContainer" containerID="117a3b16d6b5851cd1cae61ecf0c84b5e0f2f7acc73a32f5b93946d4c0c0096a"
Oct 31 13:37:08.949648 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.949623    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-xr569"]
Oct 31 13:37:08.963117 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.963092    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"]
Oct 31 13:37:08.977993 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:08.977961    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rhsf5"]
Oct 31 13:37:09.043722 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:09.043591    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"]
Oct 31 13:37:09.048161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:09.048132    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9kbfj"]
Oct 31 13:37:09.058425 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:09.058394    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"]
Oct 31 13:37:09.071557 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:09.071526    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-f7n7m"]
Oct 31 13:37:10.548780 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:10.548742    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0956d488-da8e-4597-93e5-4a6918b1887d" path="/var/lib/kubelet/pods/0956d488-da8e-4597-93e5-4a6918b1887d/volumes"
Oct 31 13:37:10.549396 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:10.549376    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d0989b-0ce8-4041-bf02-e9af6b10c3ae" path="/var/lib/kubelet/pods/11d0989b-0ce8-4041-bf02-e9af6b10c3ae/volumes"
Oct 31 13:37:10.549882 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:10.549866    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" path="/var/lib/kubelet/pods/48afff97-c9ef-4c8c-8304-9abb9ce5a0b2/volumes"
Oct 31 13:37:10.550698 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:10.550677    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" path="/var/lib/kubelet/pods/63fba834-3ec6-4542-97b5-5ddac5d4d450/volumes"
Oct 31 13:37:11.327318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.323425    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"]
Oct 31 13:37:11.327318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.324101    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" containerName="istio-proxy" containerID="cri-o://ec288aa507f384ec02bd94907cb655242611bc2827f8f5beffe333894719f204" gracePeriod=30
Oct 31 13:37:11.426320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.423147    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"]
Oct 31 13:37:11.426320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.424218    4660 cpu_manager.go:414] "containerMap: removing container" podUID="48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" containerName="istio-proxy"
Oct 31 13:37:11.426320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.424241    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" containerName="istio-proxy"
Oct 31 13:37:11.426320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.424278    4660 cpu_manager.go:414] "containerMap: removing container" podUID="0956d488-da8e-4597-93e5-4a6918b1887d" containerName="istio-proxy"
Oct 31 13:37:11.426320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.424286    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0956d488-da8e-4597-93e5-4a6918b1887d" containerName="istio-proxy"
Oct 31 13:37:11.426320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.424333    4660 cpu_manager.go:414] "containerMap: removing container" podUID="30ba0e0d-84ed-4caf-9351-a65cbf400dec" containerName="istio-proxy"
Oct 31 13:37:11.426320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.424341    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ba0e0d-84ed-4caf-9351-a65cbf400dec" containerName="istio-proxy"
Oct 31 13:37:11.426320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.424384    4660 cpu_manager.go:414] "containerMap: removing container" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy"
Oct 31 13:37:11.426320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.424390    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy"
Oct 31 13:37:11.426320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.424416    4660 cpu_manager.go:414] "containerMap: removing container" podUID="11d0989b-0ce8-4041-bf02-e9af6b10c3ae" containerName="istio-proxy"
Oct 31 13:37:11.426320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.424424    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d0989b-0ce8-4041-bf02-e9af6b10c3ae" containerName="istio-proxy"
Oct 31 13:37:11.426320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.424587    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="63fba834-3ec6-4542-97b5-5ddac5d4d450" containerName="istio-proxy"
Oct 31 13:37:11.426320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.424607    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d0989b-0ce8-4041-bf02-e9af6b10c3ae" containerName="istio-proxy"
Oct 31 13:37:11.426320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.424619    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="48afff97-c9ef-4c8c-8304-9abb9ce5a0b2" containerName="istio-proxy"
Oct 31 13:37:11.426320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.424630    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="0956d488-da8e-4597-93e5-4a6918b1887d" containerName="istio-proxy"
Oct 31 13:37:11.426320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.424641    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ba0e0d-84ed-4caf-9351-a65cbf400dec" containerName="istio-proxy"
Oct 31 13:37:11.488317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.486001    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.488317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.486810    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"]
Oct 31 13:37:11.498418 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.494579    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.504519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.504480    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"]
Oct 31 13:37:11.513969 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.510072    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"]
Oct 31 13:37:11.538608 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.537280    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.538608 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.537411    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.538608 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.537514    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.538608 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.537548    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.538608 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.537582    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.538608 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.537649    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.538608 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.537687    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bnzj\" (UniqueName: \"kubernetes.io/projected/ede7a814-d7a4-44ef-b0b4-bf7377741d13-kube-api-access-6bnzj\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.538608 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.537724    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.538608 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.537800    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.639615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.639570    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.639615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.639618    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb081f1b-a184-4656-a834-240605f9693b-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.640145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.639820    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.640145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.639971    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.640145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.640038    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.640145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.640042    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.640145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.640065    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb081f1b-a184-4656-a834-240605f9693b-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.640438 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.640173    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.640438 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.640212    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.640438 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.640249    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.640438 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.640279    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.640438 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.640330    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.640438 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.640379    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljmk5\" (UniqueName: \"kubernetes.io/projected/fb081f1b-a184-4656-a834-240605f9693b-kube-api-access-ljmk5\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.640438 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.640412    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.640782 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.640453    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bnzj\" (UniqueName: \"kubernetes.io/projected/ede7a814-d7a4-44ef-b0b4-bf7377741d13-kube-api-access-6bnzj\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.640782 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.640515    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.640782 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.640286    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.641031 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.640989    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.641089 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.641062    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb081f1b-a184-4656-a834-240605f9693b-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.641218 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.641196    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.641340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.641279    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.641473 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.641450    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.641679 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.641646    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.643227 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.643203    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.644219 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.644191    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.654642 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.654615    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.654757 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.654723    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bnzj\" (UniqueName: \"kubernetes.io/projected/ede7a814-d7a4-44ef-b0b4-bf7377741d13-kube-api-access-6bnzj\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.745087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.743834    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.745087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.743895    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.745087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.743940    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljmk5\" (UniqueName: \"kubernetes.io/projected/fb081f1b-a184-4656-a834-240605f9693b-kube-api-access-ljmk5\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.745087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.743994    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb081f1b-a184-4656-a834-240605f9693b-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.745087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.744025    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.745087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.744064    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.745087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.744125    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb081f1b-a184-4656-a834-240605f9693b-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.745087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.744380    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.745087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.744405    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb081f1b-a184-4656-a834-240605f9693b-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.748488 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.747515    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.748488 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.747756    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.748488 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.747960    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.748488 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.748433    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.751709 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.751679    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb081f1b-a184-4656-a834-240605f9693b-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.754325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.753645    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb081f1b-a184-4656-a834-240605f9693b-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.763779 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.763752    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb081f1b-a184-4656-a834-240605f9693b-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.803252 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.803122    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljmk5\" (UniqueName: \"kubernetes.io/projected/fb081f1b-a184-4656-a834-240605f9693b-kube-api-access-ljmk5\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.803453 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.803399    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:11.877407 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.867892    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:11.877407 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:11.876845    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:12.437641 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:37:12.437604    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede7a814_d7a4_44ef_b0b4_bf7377741d13.slice/crio-962e0dcae352d166000290a22260239b0a10cb4dcf29955fab37ddf6405ff0e8 WatchSource:0}: Error finding container 962e0dcae352d166000290a22260239b0a10cb4dcf29955fab37ddf6405ff0e8: Status 404 returned error can't find the container with id 962e0dcae352d166000290a22260239b0a10cb4dcf29955fab37ddf6405ff0e8
Oct 31 13:37:12.443496 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:12.442528    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"]
Oct 31 13:37:12.444814 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:12.443747    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:37:12.444814 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:12.443856    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:37:12.444814 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:12.443938    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:37:12.449099 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:12.449073    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"]
Oct 31 13:37:12.452331 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:12.452257    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:37:12.452434 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:12.452375    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:37:12.452434 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:12.452412    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:37:13.210344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:13.209964    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9" event={"ID":"fb081f1b-a184-4656-a834-240605f9693b","Type":"ContainerStarted","Data":"0c543a4a65bbfb2b89063ee6f745227e6863eadf7b0b6fc716a7f44ff86f7558"}
Oct 31 13:37:13.210344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:13.210038    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9" event={"ID":"fb081f1b-a184-4656-a834-240605f9693b","Type":"ContainerStarted","Data":"f2c27fb5d20d340d95d4a879fea238d17662d4eab745e68d0c87c0689bf49291"}
Oct 31 13:37:13.216571 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:13.215680    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt" event={"ID":"ede7a814-d7a4-44ef-b0b4-bf7377741d13","Type":"ContainerStarted","Data":"ad35002e2b3dac940ba41aff72582a11e6b5da903574f5efd1fee4f78c581b96"}
Oct 31 13:37:13.216856 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:13.216817    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt" event={"ID":"ede7a814-d7a4-44ef-b0b4-bf7377741d13","Type":"ContainerStarted","Data":"962e0dcae352d166000290a22260239b0a10cb4dcf29955fab37ddf6405ff0e8"}
Oct 31 13:37:13.286943 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:13.286886    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9" podStartSLOduration=2.286869796 podStartE2EDuration="2.286869796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:13.264627693 +0000 UTC m=+2032.883286858" watchObservedRunningTime="2026-10-31 13:37:13.286869796 +0000 UTC m=+2032.905528960"
Oct 31 13:37:13.315477 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:13.315417    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt" podStartSLOduration=2.31539524 podStartE2EDuration="2.31539524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:13.307982488 +0000 UTC m=+2032.926641653" watchObservedRunningTime="2026-10-31 13:37:13.31539524 +0000 UTC m=+2032.934054407"
Oct 31 13:37:13.872478 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:13.872350    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:13.888318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:13.877827    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:13.888318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:13.885422    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 13:37:13.888318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:13.885484    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt" podUID="ede7a814-d7a4-44ef-b0b4-bf7377741d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 13:37:13.888980 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:13.888948    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 13:37:13.889093 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:13.889012    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9" podUID="fb081f1b-a184-4656-a834-240605f9693b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 13:37:14.869338 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:14.869284    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 13:37:14.870007 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:14.869974    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt" podUID="ede7a814-d7a4-44ef-b0b4-bf7377741d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 13:37:14.885855 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:14.885699    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 13:37:14.885855 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:14.885780    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9" podUID="fb081f1b-a184-4656-a834-240605f9693b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 13:37:15.876339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:15.876013    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 13:37:15.876339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:15.876075    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt" podUID="ede7a814-d7a4-44ef-b0b4-bf7377741d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 13:37:15.880844 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:15.880809    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 13:37:15.881012 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:15.880867    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9" podUID="fb081f1b-a184-4656-a834-240605f9693b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 13:37:16.885415 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:16.875107    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 13:37:16.885415 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:16.875171    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt" podUID="ede7a814-d7a4-44ef-b0b4-bf7377741d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 13:37:16.885415 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:16.880067    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 13:37:16.885415 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:16.880126    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9" podUID="fb081f1b-a184-4656-a834-240605f9693b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 13:37:16.917275 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:16.917238    4660 generic.go:356] "Generic (PLEG): container finished" podID="c5d5ff03-7667-4572-9447-e2972398e04e" containerID="ec288aa507f384ec02bd94907cb655242611bc2827f8f5beffe333894719f204" exitCode=0
Oct 31 13:37:17.198768 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:17.198285    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" event={"ID":"c5d5ff03-7667-4572-9447-e2972398e04e","Type":"ContainerDied","Data":"ec288aa507f384ec02bd94907cb655242611bc2827f8f5beffe333894719f204"}
Oct 31 13:37:17.198768 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:17.198444    4660 scope.go:122] "RemoveContainer" containerID="3c3f876906cb56e137fa72e5d512b21b4e1bc9cedcadac824c6705e35f9ded66"
Oct 31 13:37:17.453365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:17.425204    4660 patch_prober.go:29] interesting pod/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 13:37:17.453365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:17.425268    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 13:37:17.884369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:17.882448    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 13:37:17.884369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:17.882521    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt" podUID="ede7a814-d7a4-44ef-b0b4-bf7377741d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 13:37:17.884369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:17.882979    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 13:37:17.884369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:17.883018    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9" podUID="fb081f1b-a184-4656-a834-240605f9693b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 13:37:18.095647 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.091563    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:37:18.189987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.184891    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5d5ff03-7667-4572-9447-e2972398e04e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d5ff03-7667-4572-9447-e2972398e04e-istio-podinfo\") pod \"c5d5ff03-7667-4572-9447-e2972398e04e\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") "
Oct 31 13:37:18.189987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.184967    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-istio-data\") pod \"c5d5ff03-7667-4572-9447-e2972398e04e\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") "
Oct 31 13:37:18.189987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.185105    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-credential-socket\") pod \"c5d5ff03-7667-4572-9447-e2972398e04e\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") "
Oct 31 13:37:18.189987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.185136    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d5ff03-7667-4572-9447-e2972398e04e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d5ff03-7667-4572-9447-e2972398e04e-istiod-ca-cert\") pod \"c5d5ff03-7667-4572-9447-e2972398e04e\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") "
Oct 31 13:37:18.189987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.185195    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-workload-socket\") pod \"c5d5ff03-7667-4572-9447-e2972398e04e\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") "
Oct 31 13:37:18.189987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.185346    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d5ff03-7667-4572-9447-e2972398e04e-kube-api-access-qmxn5\" (UniqueName: \"kubernetes.io/projected/c5d5ff03-7667-4572-9447-e2972398e04e-kube-api-access-qmxn5\") pod \"c5d5ff03-7667-4572-9447-e2972398e04e\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") "
Oct 31 13:37:18.189987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.185408    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-istio-envoy\") pod \"c5d5ff03-7667-4572-9447-e2972398e04e\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") "
Oct 31 13:37:18.189987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.185527    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-workload-certs\") pod \"c5d5ff03-7667-4572-9447-e2972398e04e\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") "
Oct 31 13:37:18.189987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.185643    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d5ff03-7667-4572-9447-e2972398e04e-istio-token\" (UniqueName: \"kubernetes.io/projected/c5d5ff03-7667-4572-9447-e2972398e04e-istio-token\") pod \"c5d5ff03-7667-4572-9447-e2972398e04e\" (UID: \"c5d5ff03-7667-4572-9447-e2972398e04e\") "
Oct 31 13:37:18.191651 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.191451    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-workload-socket" pod "c5d5ff03-7667-4572-9447-e2972398e04e" (UID: "c5d5ff03-7667-4572-9447-e2972398e04e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:18.191980 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.191816    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-istio-data" pod "c5d5ff03-7667-4572-9447-e2972398e04e" (UID: "c5d5ff03-7667-4572-9447-e2972398e04e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:18.192268 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.192098    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-credential-socket" pod "c5d5ff03-7667-4572-9447-e2972398e04e" (UID: "c5d5ff03-7667-4572-9447-e2972398e04e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:18.192795 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.192508    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d5ff03-7667-4572-9447-e2972398e04e-istiod-ca-cert" pod "c5d5ff03-7667-4572-9447-e2972398e04e" (UID: "c5d5ff03-7667-4572-9447-e2972398e04e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:18.198682 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.192938    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-workload-certs" pod "c5d5ff03-7667-4572-9447-e2972398e04e" (UID: "c5d5ff03-7667-4572-9447-e2972398e04e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:18.198682 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.193352    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d5ff03-7667-4572-9447-e2972398e04e-istio-token" pod "c5d5ff03-7667-4572-9447-e2972398e04e" (UID: "c5d5ff03-7667-4572-9447-e2972398e04e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:18.198682 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.195156    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5d5ff03-7667-4572-9447-e2972398e04e-istio-podinfo" pod "c5d5ff03-7667-4572-9447-e2972398e04e" (UID: "c5d5ff03-7667-4572-9447-e2972398e04e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:37:18.198682 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.198590    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-istio-envoy" pod "c5d5ff03-7667-4572-9447-e2972398e04e" (UID: "c5d5ff03-7667-4572-9447-e2972398e04e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:18.198959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.198829    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d5ff03-7667-4572-9447-e2972398e04e-kube-api-access-qmxn5" pod "c5d5ff03-7667-4572-9447-e2972398e04e" (UID: "c5d5ff03-7667-4572-9447-e2972398e04e"). InnerVolumeSpecName "kube-api-access-qmxn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:18.341150 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.310757    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:18.341150 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.310798    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmxn5\" (UniqueName: \"kubernetes.io/projected/c5d5ff03-7667-4572-9447-e2972398e04e-kube-api-access-qmxn5\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:18.341150 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.310816    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:18.341150 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.310830    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:18.341150 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.310846    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d5ff03-7667-4572-9447-e2972398e04e-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:18.341150 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.310859    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d5ff03-7667-4572-9447-e2972398e04e-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:18.341150 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.310873    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:18.341150 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.310888    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5ff03-7667-4572-9447-e2972398e04e-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:18.341150 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:18.310905    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d5ff03-7667-4572-9447-e2972398e04e-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:19.001101 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:19.000444    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 13:37:19.001101 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:19.000520    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9" podUID="fb081f1b-a184-4656-a834-240605f9693b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 13:37:19.001101 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:19.000955    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 13:37:19.001101 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:19.000990    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt" podUID="ede7a814-d7a4-44ef-b0b4-bf7377741d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 13:37:19.162754 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:19.162713    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst" event={"ID":"c5d5ff03-7667-4572-9447-e2972398e04e","Type":"ContainerDied","Data":"cad2714d521b6cae79ed9f6d47ba32ff9a85c8808af1b0daa78e5782e66fcf31"}
Oct 31 13:37:19.175570 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:19.175537    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"
Oct 31 13:37:19.177540 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:19.177514    4660 scope.go:122] "RemoveContainer" containerID="ec288aa507f384ec02bd94907cb655242611bc2827f8f5beffe333894719f204"
Oct 31 13:37:19.398807 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:19.398697    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"]
Oct 31 13:37:19.508418 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:19.442338    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cce3e7-openshift-default-7479c8855-v2jst"]
Oct 31 13:37:19.762378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:19.760467    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:37:19.877358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:19.876752    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 13:37:19.877358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:19.876839    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt" podUID="ede7a814-d7a4-44ef-b0b4-bf7377741d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 13:37:19.877640 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:19.877541    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 13:37:19.877640 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:19.877585    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9" podUID="fb081f1b-a184-4656-a834-240605f9693b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 13:37:19.979478 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:19.974151    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:37:20.004319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:19.989399    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:20.086978 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:20.086878    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:20.087593 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:20.087564    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:37:20.115456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:20.115396    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:20.275322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:20.270418    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:20.297325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:20.296416    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:37:20.451501 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:20.445391    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:37:20.598317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:20.593493    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" path="/var/lib/kubelet/pods/c5d5ff03-7667-4572-9447-e2972398e04e/volumes"
Oct 31 13:37:20.599825 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:20.598913    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:20.826370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:20.816542    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:20.826370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:20.817103    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:37:20.853331 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:37:20.850762    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:37:20.888982 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:20.888944    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 13:37:20.889273 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:20.889242    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt" podUID="ede7a814-d7a4-44ef-b0b4-bf7377741d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 13:37:20.890094 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:20.890071    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 13:37:20.890310 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:20.890231    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9" podUID="fb081f1b-a184-4656-a834-240605f9693b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 13:37:21.006318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:20.998744    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:21.006318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:20.999086    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:37:21.092049 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:21.087390    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:21.092049 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:21.087948    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:37:21.158702 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:21.140965    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 13:37:21.289930 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:21.289899    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:37:21.300343 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:21.298997    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:37:21.366587 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:21.366487    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:21.367373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:21.367348    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:37:21.398669 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:21.398515    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:37:21.890319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:21.889740    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:21.890319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:21.890245    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:21.898665 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:21.892871    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:37:21.924723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:21.922246    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:21.924723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:21.923596    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:21.926494 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:21.926344    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 13:37:24.904428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:24.901025    4660 patch_prober.go:29] interesting pod/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": read tcp 10.133.0.2:47176->10.133.0.119:15021: read: connection reset by peer" start-of-body=
Oct 31 13:37:24.904428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:24.901120    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh" podUID="30b88da6-a8a1-4882-8264-3ae18b1350a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": read tcp 10.133.0.2:47176->10.133.0.119:15021: read: connection reset by peer"
Oct 31 13:37:25.671068 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:25.671020    4660 generic.go:356] "Generic (PLEG): container finished" podID="30b88da6-a8a1-4882-8264-3ae18b1350a9" containerID="ad3f9d5af33a976432799429a354c069336a507eb64096de4eda3117c922e778" exitCode=0
Oct 31 13:37:25.674046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:25.671423    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh" event={"ID":"30b88da6-a8a1-4882-8264-3ae18b1350a9","Type":"ContainerDied","Data":"ad3f9d5af33a976432799429a354c069336a507eb64096de4eda3117c922e778"}
Oct 31 13:37:25.674046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:25.672082    4660 scope.go:122] "RemoveContainer" containerID="ad3f9d5af33a976432799429a354c069336a507eb64096de4eda3117c922e778"
Oct 31 13:37:25.677878 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:25.676807    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:37:25.677878 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:25.676963    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:37:25.677878 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:25.677059    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:37:26.682546 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:26.682505    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh" event={"ID":"30b88da6-a8a1-4882-8264-3ae18b1350a9","Type":"ContainerStarted","Data":"41b0a19464ccfc17222560df567a09d33c0801873744b12491be9bd0ca908c59"}
Oct 31 13:37:27.484498 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:27.484463    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:37:27.484901 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:27.484877    4660 patch_prober.go:29] interesting pod/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 13:37:27.485074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:27.485050    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh" podUID="30b88da6-a8a1-4882-8264-3ae18b1350a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 13:37:28.322666 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:28.322620    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:28.334159 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:28.334118    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:28.436318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:28.431329    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:28.461320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:28.458406    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:28.555324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:28.551600    4660 patch_prober.go:29] interesting pod/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 13:37:28.555324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:28.551665    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh" podUID="30b88da6-a8a1-4882-8264-3ae18b1350a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 13:37:28.572094 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:28.572044    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:29.483510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:29.483452    4660 patch_prober.go:29] interesting pod/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 13:37:29.483914 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:29.483570    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh" podUID="30b88da6-a8a1-4882-8264-3ae18b1350a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 13:37:30.483495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:30.483455    4660 patch_prober.go:29] interesting pod/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 13:37:30.483716 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:30.483546    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh" podUID="30b88da6-a8a1-4882-8264-3ae18b1350a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 13:37:31.487669 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:31.487632    4660 patch_prober.go:29] interesting pod/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 13:37:31.488283 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:31.488244    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh" podUID="30b88da6-a8a1-4882-8264-3ae18b1350a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 13:37:32.570366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:32.561114    4660 patch_prober.go:29] interesting pod/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 13:37:32.570366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:32.561186    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh" podUID="30b88da6-a8a1-4882-8264-3ae18b1350a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 13:37:33.571325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:33.570275    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:37:33.571875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:33.571480    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:37:33.572458 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:33.572434    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:37:37.754714 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:37.754634    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"]
Oct 31 13:37:37.755218 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:37.754907    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" podUID="d86a5000-770e-409d-b23a-f836beb99b08" containerName="istio-proxy" containerID="cri-o://0726287ce8f68afb6feb05e5741cae036ad6009153a1972070c985101a72a0dc" gracePeriod=30
Oct 31 13:37:39.755829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:39.755784    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"]
Oct 31 13:37:39.756408 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:39.756061    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr" podUID="c6bae42a-e94c-453a-8a9c-9da54f465a6c" containerName="maas-api" containerID="cri-o://f5c2dde697df00d8c4962b395540d3e0b097f128c4aca392fc58f7ab302d7a39" gracePeriod=30
Oct 31 13:37:40.489898 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:40.488778    4660 generic.go:356] "Generic (PLEG): container finished" podID="c6bae42a-e94c-453a-8a9c-9da54f465a6c" containerID="f5c2dde697df00d8c4962b395540d3e0b097f128c4aca392fc58f7ab302d7a39" exitCode=0
Oct 31 13:37:40.489898 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:40.488847    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr" event={"ID":"c6bae42a-e94c-453a-8a9c-9da54f465a6c","Type":"ContainerDied","Data":"f5c2dde697df00d8c4962b395540d3e0b097f128c4aca392fc58f7ab302d7a39"}
Oct 31 13:37:40.536353 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:40.536323    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:37:40.695493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:40.695409    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6bae42a-e94c-453a-8a9c-9da54f465a6c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6bae42a-e94c-453a-8a9c-9da54f465a6c-maas-api-tls\") pod \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\" (UID: \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\") "
Oct 31 13:37:40.695646 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:40.695548    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6bae42a-e94c-453a-8a9c-9da54f465a6c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6bae42a-e94c-453a-8a9c-9da54f465a6c-openshift-service-ca\") pod \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\" (UID: \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\") "
Oct 31 13:37:40.695646 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:40.695640    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6bae42a-e94c-453a-8a9c-9da54f465a6c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6bae42a-e94c-453a-8a9c-9da54f465a6c-metrics-certs\") pod \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\" (UID: \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\") "
Oct 31 13:37:40.695735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:40.695689    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6bae42a-e94c-453a-8a9c-9da54f465a6c-kube-api-access-6vlql\" (UniqueName: \"kubernetes.io/projected/c6bae42a-e94c-453a-8a9c-9da54f465a6c-kube-api-access-6vlql\") pod \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\" (UID: \"c6bae42a-e94c-453a-8a9c-9da54f465a6c\") "
Oct 31 13:37:40.695927 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:40.695891    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6bae42a-e94c-453a-8a9c-9da54f465a6c-openshift-service-ca" pod "c6bae42a-e94c-453a-8a9c-9da54f465a6c" (UID: "c6bae42a-e94c-453a-8a9c-9da54f465a6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:40.696528 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:40.696504    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6bae42a-e94c-453a-8a9c-9da54f465a6c-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:40.697841 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:40.697807    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6bae42a-e94c-453a-8a9c-9da54f465a6c-maas-api-tls" pod "c6bae42a-e94c-453a-8a9c-9da54f465a6c" (UID: "c6bae42a-e94c-453a-8a9c-9da54f465a6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:37:40.697841 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:40.697827    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6bae42a-e94c-453a-8a9c-9da54f465a6c-metrics-certs" pod "c6bae42a-e94c-453a-8a9c-9da54f465a6c" (UID: "c6bae42a-e94c-453a-8a9c-9da54f465a6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:37:40.697984 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:40.697897    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6bae42a-e94c-453a-8a9c-9da54f465a6c-kube-api-access-6vlql" pod "c6bae42a-e94c-453a-8a9c-9da54f465a6c" (UID: "c6bae42a-e94c-453a-8a9c-9da54f465a6c"). InnerVolumeSpecName "kube-api-access-6vlql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:40.798800 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:40.798764    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6bae42a-e94c-453a-8a9c-9da54f465a6c-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:40.798800 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:40.798796    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vlql\" (UniqueName: \"kubernetes.io/projected/c6bae42a-e94c-453a-8a9c-9da54f465a6c-kube-api-access-6vlql\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:40.799223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:40.798814    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6bae42a-e94c-453a-8a9c-9da54f465a6c-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:41.556316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:41.551237    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr" event={"ID":"c6bae42a-e94c-453a-8a9c-9da54f465a6c","Type":"ContainerDied","Data":"f5c6cb863edc1cd6be9cbb98c955a87931bd297d6649b78137028771f5146ff0"}
Oct 31 13:37:41.556316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:41.551347    4660 scope.go:122] "RemoveContainer" containerID="f5c2dde697df00d8c4962b395540d3e0b097f128c4aca392fc58f7ab302d7a39"
Oct 31 13:37:41.556605 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:41.556494    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"
Oct 31 13:37:41.829287 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:41.826134    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"]
Oct 31 13:37:41.874873 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:41.874834    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6aa924-64b9c5c4c6-fgnrr"]
Oct 31 13:37:42.539496 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:42.539454    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6bae42a-e94c-453a-8a9c-9da54f465a6c" path="/var/lib/kubelet/pods/c6bae42a-e94c-453a-8a9c-9da54f465a6c/volumes"
Oct 31 13:37:43.538091 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.535766    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:37:43.590364 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.583058    4660 generic.go:356] "Generic (PLEG): container finished" podID="d86a5000-770e-409d-b23a-f836beb99b08" containerID="0726287ce8f68afb6feb05e5741cae036ad6009153a1972070c985101a72a0dc" exitCode=0
Oct 31 13:37:43.590364 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.583169    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" event={"ID":"d86a5000-770e-409d-b23a-f836beb99b08","Type":"ContainerDied","Data":"0726287ce8f68afb6feb05e5741cae036ad6009153a1972070c985101a72a0dc"}
Oct 31 13:37:43.590364 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.583214    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj" event={"ID":"d86a5000-770e-409d-b23a-f836beb99b08","Type":"ContainerDied","Data":"72fb24090d9f616cabc08ef15cc765fee8140089419130eb694787619758a739"}
Oct 31 13:37:43.590364 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.583236    4660 scope.go:122] "RemoveContainer" containerID="0726287ce8f68afb6feb05e5741cae036ad6009153a1972070c985101a72a0dc"
Oct 31 13:37:43.590364 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.583382    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"
Oct 31 13:37:43.616369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.616183    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d86a5000-770e-409d-b23a-f836beb99b08-kube-api-access-9smhk\" (UniqueName: \"kubernetes.io/projected/d86a5000-770e-409d-b23a-f836beb99b08-kube-api-access-9smhk\") pod \"d86a5000-770e-409d-b23a-f836beb99b08\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") "
Oct 31 13:37:43.616369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.616266    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-workload-socket\") pod \"d86a5000-770e-409d-b23a-f836beb99b08\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") "
Oct 31 13:37:43.616369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.616356    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d86a5000-770e-409d-b23a-f836beb99b08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86a5000-770e-409d-b23a-f836beb99b08-istiod-ca-cert\") pod \"d86a5000-770e-409d-b23a-f836beb99b08\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") "
Oct 31 13:37:43.616679 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.616394    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-workload-certs\") pod \"d86a5000-770e-409d-b23a-f836beb99b08\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") "
Oct 31 13:37:43.616679 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.616486    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-istio-data\") pod \"d86a5000-770e-409d-b23a-f836beb99b08\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") "
Oct 31 13:37:43.616679 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.616594    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-istio-envoy\") pod \"d86a5000-770e-409d-b23a-f836beb99b08\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") "
Oct 31 13:37:43.616679 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.616650    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-credential-socket\") pod \"d86a5000-770e-409d-b23a-f836beb99b08\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") "
Oct 31 13:37:43.616876 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.616769    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d86a5000-770e-409d-b23a-f836beb99b08-istio-token\" (UniqueName: \"kubernetes.io/projected/d86a5000-770e-409d-b23a-f836beb99b08-istio-token\") pod \"d86a5000-770e-409d-b23a-f836beb99b08\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") "
Oct 31 13:37:43.616876 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.616819    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d86a5000-770e-409d-b23a-f836beb99b08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86a5000-770e-409d-b23a-f836beb99b08-istio-podinfo\") pod \"d86a5000-770e-409d-b23a-f836beb99b08\" (UID: \"d86a5000-770e-409d-b23a-f836beb99b08\") "
Oct 31 13:37:43.619565 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.617914    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-workload-socket" pod "d86a5000-770e-409d-b23a-f836beb99b08" (UID: "d86a5000-770e-409d-b23a-f836beb99b08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:43.624484 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.623385    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-istio-data" pod "d86a5000-770e-409d-b23a-f836beb99b08" (UID: "d86a5000-770e-409d-b23a-f836beb99b08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:43.624484 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.623739    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86a5000-770e-409d-b23a-f836beb99b08-istiod-ca-cert" pod "d86a5000-770e-409d-b23a-f836beb99b08" (UID: "d86a5000-770e-409d-b23a-f836beb99b08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:43.631137 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.627332    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-credential-socket" pod "d86a5000-770e-409d-b23a-f836beb99b08" (UID: "d86a5000-770e-409d-b23a-f836beb99b08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:43.631137 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.627665    4660 scope.go:122] "RemoveContainer" containerID="26aaf65ec927aef941594f84586d17582de30dd442b3920221eb17f2babc4506"
Oct 31 13:37:43.631137 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.630083    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86a5000-770e-409d-b23a-f836beb99b08-istio-token" pod "d86a5000-770e-409d-b23a-f836beb99b08" (UID: "d86a5000-770e-409d-b23a-f836beb99b08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:43.639598 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.632489    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-workload-certs" pod "d86a5000-770e-409d-b23a-f836beb99b08" (UID: "d86a5000-770e-409d-b23a-f836beb99b08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:43.639598 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.636497    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d86a5000-770e-409d-b23a-f836beb99b08-istio-podinfo" pod "d86a5000-770e-409d-b23a-f836beb99b08" (UID: "d86a5000-770e-409d-b23a-f836beb99b08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:37:43.640916 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.639888    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86a5000-770e-409d-b23a-f836beb99b08-kube-api-access-9smhk" pod "d86a5000-770e-409d-b23a-f836beb99b08" (UID: "d86a5000-770e-409d-b23a-f836beb99b08"). InnerVolumeSpecName "kube-api-access-9smhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:43.642519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.642491    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-istio-envoy" pod "d86a5000-770e-409d-b23a-f836beb99b08" (UID: "d86a5000-770e-409d-b23a-f836beb99b08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:43.692238 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.691453    4660 scope.go:122] "RemoveContainer" containerID="0726287ce8f68afb6feb05e5741cae036ad6009153a1972070c985101a72a0dc"
Oct 31 13:37:43.692541 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:37:43.692336    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0726287ce8f68afb6feb05e5741cae036ad6009153a1972070c985101a72a0dc\": container with ID starting with 0726287ce8f68afb6feb05e5741cae036ad6009153a1972070c985101a72a0dc not found: ID does not exist" containerID="0726287ce8f68afb6feb05e5741cae036ad6009153a1972070c985101a72a0dc"
Oct 31 13:37:43.692541 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.692378    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0726287ce8f68afb6feb05e5741cae036ad6009153a1972070c985101a72a0dc"} err="failed to get container status \"0726287ce8f68afb6feb05e5741cae036ad6009153a1972070c985101a72a0dc\": rpc error: code = NotFound desc = could not find container \"0726287ce8f68afb6feb05e5741cae036ad6009153a1972070c985101a72a0dc\": container with ID starting with 0726287ce8f68afb6feb05e5741cae036ad6009153a1972070c985101a72a0dc not found: ID does not exist"
Oct 31 13:37:43.692541 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.692405    4660 scope.go:122] "RemoveContainer" containerID="26aaf65ec927aef941594f84586d17582de30dd442b3920221eb17f2babc4506"
Oct 31 13:37:43.692872 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:37:43.692775    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26aaf65ec927aef941594f84586d17582de30dd442b3920221eb17f2babc4506\": container with ID starting with 26aaf65ec927aef941594f84586d17582de30dd442b3920221eb17f2babc4506 not found: ID does not exist" containerID="26aaf65ec927aef941594f84586d17582de30dd442b3920221eb17f2babc4506"
Oct 31 13:37:43.692872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.692805    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26aaf65ec927aef941594f84586d17582de30dd442b3920221eb17f2babc4506"} err="failed to get container status \"26aaf65ec927aef941594f84586d17582de30dd442b3920221eb17f2babc4506\": rpc error: code = NotFound desc = could not find container \"26aaf65ec927aef941594f84586d17582de30dd442b3920221eb17f2babc4506\": container with ID starting with 26aaf65ec927aef941594f84586d17582de30dd442b3920221eb17f2babc4506 not found: ID does not exist"
Oct 31 13:37:43.740955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.731066    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:43.740955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.731102    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:43.740955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.731118    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:43.740955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.731133    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d86a5000-770e-409d-b23a-f836beb99b08-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:43.740955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.731149    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86a5000-770e-409d-b23a-f836beb99b08-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:43.740955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.731164    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9smhk\" (UniqueName: \"kubernetes.io/projected/d86a5000-770e-409d-b23a-f836beb99b08-kube-api-access-9smhk\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:43.740955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.731183    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:43.740955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.731197    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86a5000-770e-409d-b23a-f836beb99b08-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:43.740955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:43.731210    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86a5000-770e-409d-b23a-f836beb99b08-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:44.157322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:44.154061    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"]
Oct 31 13:37:44.162098 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:44.162047    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-580f24-openshift-default-86d4586cbc-57kvj"]
Oct 31 13:37:44.531687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:44.531650    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d86a5000-770e-409d-b23a-f836beb99b08" path="/var/lib/kubelet/pods/d86a5000-770e-409d-b23a-f836beb99b08/volumes"
Oct 31 13:37:49.813331 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.812463    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"]
Oct 31 13:37:49.813852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.813573    4660 cpu_manager.go:414] "containerMap: removing container" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" containerName="istio-proxy"
Oct 31 13:37:49.813852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.813596    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" containerName="istio-proxy"
Oct 31 13:37:49.813852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.813622    4660 cpu_manager.go:414] "containerMap: removing container" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" containerName="istio-proxy"
Oct 31 13:37:49.813852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.813630    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" containerName="istio-proxy"
Oct 31 13:37:49.813852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.813644    4660 cpu_manager.go:414] "containerMap: removing container" podUID="d86a5000-770e-409d-b23a-f836beb99b08" containerName="istio-proxy"
Oct 31 13:37:49.813852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.813651    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d86a5000-770e-409d-b23a-f836beb99b08" containerName="istio-proxy"
Oct 31 13:37:49.813852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.813688    4660 cpu_manager.go:414] "containerMap: removing container" podUID="d86a5000-770e-409d-b23a-f836beb99b08" containerName="istio-proxy"
Oct 31 13:37:49.813852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.813696    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d86a5000-770e-409d-b23a-f836beb99b08" containerName="istio-proxy"
Oct 31 13:37:49.813852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.813729    4660 cpu_manager.go:414] "containerMap: removing container" podUID="c6bae42a-e94c-453a-8a9c-9da54f465a6c" containerName="maas-api"
Oct 31 13:37:49.813852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.813736    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6bae42a-e94c-453a-8a9c-9da54f465a6c" containerName="maas-api"
Oct 31 13:37:49.814433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.813896    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="d86a5000-770e-409d-b23a-f836beb99b08" containerName="istio-proxy"
Oct 31 13:37:49.814433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.813911    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6bae42a-e94c-453a-8a9c-9da54f465a6c" containerName="maas-api"
Oct 31 13:37:49.814433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.813926    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" containerName="istio-proxy"
Oct 31 13:37:49.814433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.813936    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d5ff03-7667-4572-9447-e2972398e04e" containerName="istio-proxy"
Oct 31 13:37:49.922008 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.921969    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"]
Oct 31 13:37:49.922379 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:49.922358    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:37:50.058564 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:50.058528    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19742f9a-ed37-4745-a0ac-af1d842a334d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q\" (UID: \"19742f9a-ed37-4745-a0ac-af1d842a334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:37:50.075499 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:50.075397    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19742f9a-ed37-4745-a0ac-af1d842a334d-maas-api-tls\") pod \"maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q\" (UID: \"19742f9a-ed37-4745-a0ac-af1d842a334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:37:50.076195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:50.076166    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19742f9a-ed37-4745-a0ac-af1d842a334d-metrics-certs\") pod \"maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q\" (UID: \"19742f9a-ed37-4745-a0ac-af1d842a334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:37:50.080471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:50.080438    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxlf5\" (UniqueName: \"kubernetes.io/projected/19742f9a-ed37-4745-a0ac-af1d842a334d-kube-api-access-qxlf5\") pod \"maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q\" (UID: \"19742f9a-ed37-4745-a0ac-af1d842a334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:37:50.099171 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:50.090495    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-62778a-serving-cert\""
Oct 31 13:37:50.199401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:50.199356    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19742f9a-ed37-4745-a0ac-af1d842a334d-metrics-certs\") pod \"maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q\" (UID: \"19742f9a-ed37-4745-a0ac-af1d842a334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:37:50.199581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:50.199484    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxlf5\" (UniqueName: \"kubernetes.io/projected/19742f9a-ed37-4745-a0ac-af1d842a334d-kube-api-access-qxlf5\") pod \"maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q\" (UID: \"19742f9a-ed37-4745-a0ac-af1d842a334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:37:50.199581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:50.199521    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19742f9a-ed37-4745-a0ac-af1d842a334d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q\" (UID: \"19742f9a-ed37-4745-a0ac-af1d842a334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:37:50.199581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:50.199552    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19742f9a-ed37-4745-a0ac-af1d842a334d-maas-api-tls\") pod \"maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q\" (UID: \"19742f9a-ed37-4745-a0ac-af1d842a334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:37:50.200286 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:50.200260    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19742f9a-ed37-4745-a0ac-af1d842a334d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q\" (UID: \"19742f9a-ed37-4745-a0ac-af1d842a334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:37:50.218385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:50.215918    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19742f9a-ed37-4745-a0ac-af1d842a334d-maas-api-tls\") pod \"maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q\" (UID: \"19742f9a-ed37-4745-a0ac-af1d842a334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:37:50.240947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:50.240906    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19742f9a-ed37-4745-a0ac-af1d842a334d-metrics-certs\") pod \"maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q\" (UID: \"19742f9a-ed37-4745-a0ac-af1d842a334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:37:50.249113 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:50.249078    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxlf5\" (UniqueName: \"kubernetes.io/projected/19742f9a-ed37-4745-a0ac-af1d842a334d-kube-api-access-qxlf5\") pod \"maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q\" (UID: \"19742f9a-ed37-4745-a0ac-af1d842a334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:37:50.462324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:50.462213    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:37:51.145684 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:51.145647    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"]
Oct 31 13:37:51.190094 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:37:51.190061    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19742f9a_ed37_4745_a0ac_af1d842a334d.slice/crio-a96d3981b0f2d788f70e525caf15866ae3c2c6214170980b41fd91eac01128e2 WatchSource:0}: Error finding container a96d3981b0f2d788f70e525caf15866ae3c2c6214170980b41fd91eac01128e2: Status 404 returned error can't find the container with id a96d3981b0f2d788f70e525caf15866ae3c2c6214170980b41fd91eac01128e2
Oct 31 13:37:51.444665 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:37:51.444562    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:37:51.644279 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:51.634211    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"]
Oct 31 13:37:51.644279 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:51.634495    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy" containerID="cri-o://ca260e3a115d4f91f21f6219133cf4ba08eda93caa93c8823009b7d107850219" gracePeriod=30
Oct 31 13:37:52.184318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:52.171438    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q" event={"ID":"19742f9a-ed37-4745-a0ac-af1d842a334d","Type":"ContainerStarted","Data":"95c465630ac63352b83a99ba31ea4c5f82b9d68a7a8fe2f2ea5feddb9942af1b"}
Oct 31 13:37:52.184318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:52.171504    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q" event={"ID":"19742f9a-ed37-4745-a0ac-af1d842a334d","Type":"ContainerStarted","Data":"a96d3981b0f2d788f70e525caf15866ae3c2c6214170980b41fd91eac01128e2"}
Oct 31 13:37:52.184318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:52.172728    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:37:52.266079 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:52.259180    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q" podStartSLOduration=2.519887159 podStartE2EDuration="3.259158705s" totalImagesPullingTime="739.271546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:52.220230882 +0000 UTC m=+2071.838890058" watchObservedRunningTime="2026-10-31 13:37:52.259158705 +0000 UTC m=+2071.877817870"
Oct 31 13:37:52.903236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:52.903203    4660 patch_prober.go:29] interesting pod/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:37:52.903457 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:52.903261    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:37:57.221705 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.221663    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:37:57.234183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.234148    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-workload-certs\") pod \"6249baed-5e62-4178-a9a8-ffe554407c0b\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") "
Oct 31 13:37:57.234389 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.234261    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6249baed-5e62-4178-a9a8-ffe554407c0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6249baed-5e62-4178-a9a8-ffe554407c0b-istiod-ca-cert\") pod \"6249baed-5e62-4178-a9a8-ffe554407c0b\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") "
Oct 31 13:37:57.234389 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.234335    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-credential-socket\") pod \"6249baed-5e62-4178-a9a8-ffe554407c0b\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") "
Oct 31 13:37:57.234553 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.234477    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-workload-certs" pod "6249baed-5e62-4178-a9a8-ffe554407c0b" (UID: "6249baed-5e62-4178-a9a8-ffe554407c0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:57.234632 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.234604    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-credential-socket" pod "6249baed-5e62-4178-a9a8-ffe554407c0b" (UID: "6249baed-5e62-4178-a9a8-ffe554407c0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:57.234632 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.234615    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-workload-socket\") pod \"6249baed-5e62-4178-a9a8-ffe554407c0b\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") "
Oct 31 13:37:57.234746 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.234662    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-podinfo\") pod \"6249baed-5e62-4178-a9a8-ffe554407c0b\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") "
Oct 31 13:37:57.234746 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.234724    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6249baed-5e62-4178-a9a8-ffe554407c0b-kube-api-access-zsd4x\" (UniqueName: \"kubernetes.io/projected/6249baed-5e62-4178-a9a8-ffe554407c0b-kube-api-access-zsd4x\") pod \"6249baed-5e62-4178-a9a8-ffe554407c0b\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") "
Oct 31 13:37:57.234860 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.234769    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-workload-socket" pod "6249baed-5e62-4178-a9a8-ffe554407c0b" (UID: "6249baed-5e62-4178-a9a8-ffe554407c0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:57.234987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.234967    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-token\" (UniqueName: \"kubernetes.io/projected/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-token\") pod \"6249baed-5e62-4178-a9a8-ffe554407c0b\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") "
Oct 31 13:37:57.235058 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.235043    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-envoy\") pod \"6249baed-5e62-4178-a9a8-ffe554407c0b\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") "
Oct 31 13:37:57.235115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.235094    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-data\") pod \"6249baed-5e62-4178-a9a8-ffe554407c0b\" (UID: \"6249baed-5e62-4178-a9a8-ffe554407c0b\") "
Oct 31 13:37:57.235781 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.235754    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-data" pod "6249baed-5e62-4178-a9a8-ffe554407c0b" (UID: "6249baed-5e62-4178-a9a8-ffe554407c0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:57.236841 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.236820    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:57.236968 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.236844    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:57.236968 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.236861    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:57.236968 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.236874    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:57.237117 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.237043    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6249baed-5e62-4178-a9a8-ffe554407c0b-istiod-ca-cert" pod "6249baed-5e62-4178-a9a8-ffe554407c0b" (UID: "6249baed-5e62-4178-a9a8-ffe554407c0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:57.238549 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.238524    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-podinfo" pod "6249baed-5e62-4178-a9a8-ffe554407c0b" (UID: "6249baed-5e62-4178-a9a8-ffe554407c0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:37:57.238691 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.238552    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-token" pod "6249baed-5e62-4178-a9a8-ffe554407c0b" (UID: "6249baed-5e62-4178-a9a8-ffe554407c0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:57.238691 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.238670    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6249baed-5e62-4178-a9a8-ffe554407c0b-kube-api-access-zsd4x" pod "6249baed-5e62-4178-a9a8-ffe554407c0b" (UID: "6249baed-5e62-4178-a9a8-ffe554407c0b"). InnerVolumeSpecName "kube-api-access-zsd4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:57.240423 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.240398    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-envoy" pod "6249baed-5e62-4178-a9a8-ffe554407c0b" (UID: "6249baed-5e62-4178-a9a8-ffe554407c0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:57.246916 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.246434    4660 generic.go:356] "Generic (PLEG): container finished" podID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerID="ca260e3a115d4f91f21f6219133cf4ba08eda93caa93c8823009b7d107850219" exitCode=0
Oct 31 13:37:57.246916 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.246498    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"
Oct 31 13:37:57.246916 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.246647    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" event={"ID":"6249baed-5e62-4178-a9a8-ffe554407c0b","Type":"ContainerDied","Data":"ca260e3a115d4f91f21f6219133cf4ba08eda93caa93c8823009b7d107850219"}
Oct 31 13:37:57.246916 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.246715    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq" event={"ID":"6249baed-5e62-4178-a9a8-ffe554407c0b","Type":"ContainerDied","Data":"a6e2db0b5fa2e9b8f58a788e2535b7d1579df87de8422c7fb67c758016a9e3af"}
Oct 31 13:37:57.246916 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.246757    4660 scope.go:122] "RemoveContainer" containerID="ca260e3a115d4f91f21f6219133cf4ba08eda93caa93c8823009b7d107850219"
Oct 31 13:37:57.283279 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.283247    4660 scope.go:122] "RemoveContainer" containerID="9d9a900557fc97ec5245a5cd81af8edb12e443771052545a7076b9191d5cffd8"
Oct 31 13:37:57.313925 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.313892    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"]
Oct 31 13:37:57.318118 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.318089    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6aa924-openshift-default-56b6b4c99-9cxhq"]
Oct 31 13:37:57.338353 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.338324    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:57.338353 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.338346    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:57.338353 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.338356    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6249baed-5e62-4178-a9a8-ffe554407c0b-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:57.338606 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.338365    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6249baed-5e62-4178-a9a8-ffe554407c0b-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:57.338606 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.338374    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsd4x\" (UniqueName: \"kubernetes.io/projected/6249baed-5e62-4178-a9a8-ffe554407c0b-kube-api-access-zsd4x\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:57.350558 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.350537    4660 scope.go:122] "RemoveContainer" containerID="ca260e3a115d4f91f21f6219133cf4ba08eda93caa93c8823009b7d107850219"
Oct 31 13:37:57.350886 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:37:57.350862    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca260e3a115d4f91f21f6219133cf4ba08eda93caa93c8823009b7d107850219\": container with ID starting with ca260e3a115d4f91f21f6219133cf4ba08eda93caa93c8823009b7d107850219 not found: ID does not exist" containerID="ca260e3a115d4f91f21f6219133cf4ba08eda93caa93c8823009b7d107850219"
Oct 31 13:37:57.350971 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.350895    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca260e3a115d4f91f21f6219133cf4ba08eda93caa93c8823009b7d107850219"} err="failed to get container status \"ca260e3a115d4f91f21f6219133cf4ba08eda93caa93c8823009b7d107850219\": rpc error: code = NotFound desc = could not find container \"ca260e3a115d4f91f21f6219133cf4ba08eda93caa93c8823009b7d107850219\": container with ID starting with ca260e3a115d4f91f21f6219133cf4ba08eda93caa93c8823009b7d107850219 not found: ID does not exist"
Oct 31 13:37:57.350971 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.350915    4660 scope.go:122] "RemoveContainer" containerID="9d9a900557fc97ec5245a5cd81af8edb12e443771052545a7076b9191d5cffd8"
Oct 31 13:37:57.351205 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:37:57.351189    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9a900557fc97ec5245a5cd81af8edb12e443771052545a7076b9191d5cffd8\": container with ID starting with 9d9a900557fc97ec5245a5cd81af8edb12e443771052545a7076b9191d5cffd8 not found: ID does not exist" containerID="9d9a900557fc97ec5245a5cd81af8edb12e443771052545a7076b9191d5cffd8"
Oct 31 13:37:57.351270 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:57.351214    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9a900557fc97ec5245a5cd81af8edb12e443771052545a7076b9191d5cffd8"} err="failed to get container status \"9d9a900557fc97ec5245a5cd81af8edb12e443771052545a7076b9191d5cffd8\": rpc error: code = NotFound desc = could not find container \"9d9a900557fc97ec5245a5cd81af8edb12e443771052545a7076b9191d5cffd8\": container with ID starting with 9d9a900557fc97ec5245a5cd81af8edb12e443771052545a7076b9191d5cffd8 not found: ID does not exist"
Oct 31 13:37:58.529708 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:58.529666    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" path="/var/lib/kubelet/pods/6249baed-5e62-4178-a9a8-ffe554407c0b/volumes"
Oct 31 13:37:59.254666 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:37:59.254638    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:38:03.056471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.056433    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"]
Oct 31 13:38:03.057659 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.057631    4660 cpu_manager.go:414] "containerMap: removing container" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy"
Oct 31 13:38:03.057659 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.057659    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy"
Oct 31 13:38:03.057856 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.057692    4660 cpu_manager.go:414] "containerMap: removing container" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy"
Oct 31 13:38:03.057856 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.057707    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy"
Oct 31 13:38:03.057963 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.057861    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy"
Oct 31 13:38:03.057963 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.057880    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="6249baed-5e62-4178-a9a8-ffe554407c0b" containerName="istio-proxy"
Oct 31 13:38:03.057963 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.057892    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="d86a5000-770e-409d-b23a-f836beb99b08" containerName="istio-proxy"
Oct 31 13:38:03.064810 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.063932    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:38:03.073430 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.073385    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d4b52a-serving-cert\""
Oct 31 13:38:03.079717 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.078397    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"]
Oct 31 13:38:03.154259 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.154225    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7f58a2-4f5a-4100-baeb-3d0664b21972-metrics-certs\") pod \"maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7\" (UID: \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:38:03.154483 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.154346    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7f58a2-4f5a-4100-baeb-3d0664b21972-maas-api-tls\") pod \"maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7\" (UID: \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:38:03.154583 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.154560    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqwgx\" (UniqueName: \"kubernetes.io/projected/5d7f58a2-4f5a-4100-baeb-3d0664b21972-kube-api-access-kqwgx\") pod \"maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7\" (UID: \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:38:03.154773 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.154753    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7f58a2-4f5a-4100-baeb-3d0664b21972-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7\" (UID: \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:38:03.256787 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.256737    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqwgx\" (UniqueName: \"kubernetes.io/projected/5d7f58a2-4f5a-4100-baeb-3d0664b21972-kube-api-access-kqwgx\") pod \"maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7\" (UID: \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:38:03.256975 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.256933    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7f58a2-4f5a-4100-baeb-3d0664b21972-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7\" (UID: \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:38:03.257041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.257023    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7f58a2-4f5a-4100-baeb-3d0664b21972-metrics-certs\") pod \"maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7\" (UID: \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:38:03.257123 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.257107    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7f58a2-4f5a-4100-baeb-3d0664b21972-maas-api-tls\") pod \"maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7\" (UID: \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:38:03.257732 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.257707    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7f58a2-4f5a-4100-baeb-3d0664b21972-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7\" (UID: \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:38:03.259680 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.259659    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7f58a2-4f5a-4100-baeb-3d0664b21972-metrics-certs\") pod \"maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7\" (UID: \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:38:03.259834 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.259812    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7f58a2-4f5a-4100-baeb-3d0664b21972-maas-api-tls\") pod \"maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7\" (UID: \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:38:03.271151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.270753    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqwgx\" (UniqueName: \"kubernetes.io/projected/5d7f58a2-4f5a-4100-baeb-3d0664b21972-kube-api-access-kqwgx\") pod \"maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7\" (UID: \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:38:03.387673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.387646    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:38:03.928697 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:03.928659    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"]
Oct 31 13:38:03.930851 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:38:03.930637    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7f58a2_4f5a_4100_baeb_3d0664b21972.slice/crio-ea39e20d508dd6a2bb9e42e97886be8a2c5a8fd5e0cc11fbc14c94b1363442e0 WatchSource:0}: Error finding container ea39e20d508dd6a2bb9e42e97886be8a2c5a8fd5e0cc11fbc14c94b1363442e0: Status 404 returned error can't find the container with id ea39e20d508dd6a2bb9e42e97886be8a2c5a8fd5e0cc11fbc14c94b1363442e0
Oct 31 13:38:04.322310 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:04.321263    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7" event={"ID":"5d7f58a2-4f5a-4100-baeb-3d0664b21972","Type":"ContainerStarted","Data":"ea39e20d508dd6a2bb9e42e97886be8a2c5a8fd5e0cc11fbc14c94b1363442e0"}
Oct 31 13:38:05.332667 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:05.332624    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7" event={"ID":"5d7f58a2-4f5a-4100-baeb-3d0664b21972","Type":"ContainerStarted","Data":"c18a2fcee4b30525346897dc9c44f9022c4e78f2e750492a259c4c1d7b620c52"}
Oct 31 13:38:05.333100 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:05.332734    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:38:05.359039 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:05.358990    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7" podStartSLOduration=1.9084532360000002 podStartE2EDuration="2.358974589s" totalImagesPullingTime="450.521353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:05.354282316 +0000 UTC m=+2084.972941482" watchObservedRunningTime="2026-10-31 13:38:05.358974589 +0000 UTC m=+2084.977633752"
Oct 31 13:38:11.353992 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:11.353952    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:38:16.190083 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.190047    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"]
Oct 31 13:38:16.331887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.331819    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.341196 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.341145    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-61f295-openshift-default-dockercfg-8kmlw\""
Oct 31 13:38:16.439317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.438526    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-token\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.439317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.438636    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-data\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.439317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.438684    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-envoy\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.439317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.438758    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-credential-socket\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.439317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.438889    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-podinfo\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.439317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.438930    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcj44\" (UniqueName: \"kubernetes.io/projected/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-kube-api-access-lcj44\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.439317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.438972    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-workload-certs\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.439317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.439054    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istiod-ca-cert\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.439317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.439092    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-workload-socket\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.464400 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.460836    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"]
Oct 31 13:38:16.555315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.555029    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-token\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.555315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.555118    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-data\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.555315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.555164    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-envoy\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.555315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.555255    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-credential-socket\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.558245 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.556778    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-data\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.558245 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.557709    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-credential-socket\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.577311 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.565272    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-podinfo\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.577311 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.565356    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcj44\" (UniqueName: \"kubernetes.io/projected/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-kube-api-access-lcj44\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.577311 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.565421    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-workload-certs\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.577311 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.566279    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istiod-ca-cert\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.577311 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.566346    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-workload-socket\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.577311 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.567167    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-workload-socket\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.582807 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.582068    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-workload-certs\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.582807 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.582755    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istiod-ca-cert\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.618370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.613816    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-podinfo\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.623313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.619835    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-envoy\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.646387 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.644814    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-token\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.646387 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.645803    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcj44\" (UniqueName: \"kubernetes.io/projected/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-kube-api-access-lcj44\") pod \"e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") " pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:16.781622 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:16.781532    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:17.129062 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.128984    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"]
Oct 31 13:38:17.143042 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.143004    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:17.143171 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.143078    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:17.143171 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.143110    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:17.570856 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.570773    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"]
Oct 31 13:38:17.600142 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.600107    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"]
Oct 31 13:38:17.600400 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.600129    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.608146 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.608119    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4fde10-openshift-default-dockercfg-kw42g\""
Oct 31 13:38:17.700376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.700268    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-credential-socket\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.700581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.700410    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-podinfo\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.700581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.700442    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9s6k\" (UniqueName: \"kubernetes.io/projected/8a52dcc2-c2ab-430c-988e-d3b967f833f3-kube-api-access-l9s6k\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.700581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.700486    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-envoy\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.700581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.700509    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-data\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.700581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.700540    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istiod-ca-cert\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.701937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.700726    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-token\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.701937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.700835    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-workload-socket\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.701937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.700916    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-workload-certs\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.752360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.747695    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr" event={"ID":"d4e738ae-b6fd-4c4b-9413-0497e07cbc03","Type":"ContainerStarted","Data":"1198ab63c389eb8eae6823862d0dd42d6609d985f42eb1d1e591615c7e77060e"}
Oct 31 13:38:17.752360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.747763    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr" event={"ID":"d4e738ae-b6fd-4c4b-9413-0497e07cbc03","Type":"ContainerStarted","Data":"90cf10152ce03bb3f08346a891e24b4195569b413310e5d073ffb81499fb88d6"}
Oct 31 13:38:17.785584 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.785540    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:17.803275 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.803234    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-token\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.803868 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.803840    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-workload-socket\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.804107 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.804089    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-workload-certs\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.805231 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.805207    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-credential-socket\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.805468 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.805435    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-podinfo\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.805605 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.805590    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9s6k\" (UniqueName: \"kubernetes.io/projected/8a52dcc2-c2ab-430c-988e-d3b967f833f3-kube-api-access-l9s6k\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.805785 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.805768    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-envoy\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.805902 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.805886    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-data\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.806329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.806286    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-data\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.809602 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.804795    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-workload-certs\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.809602 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.807342    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istiod-ca-cert\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.809602 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.808031    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istiod-ca-cert\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.809602 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.808549    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-credential-socket\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.809602 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.805033    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-workload-socket\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.812617 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.812570    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-podinfo\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.832365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.825823    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-token\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.832365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.826042    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9s6k\" (UniqueName: \"kubernetes.io/projected/8a52dcc2-c2ab-430c-988e-d3b967f833f3-kube-api-access-l9s6k\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.843892 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.843861    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-envoy\") pod \"e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") " pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:17.958315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:17.955899    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:18.595668 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:18.591879    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr" podStartSLOduration=2.5918597610000003 podStartE2EDuration="2.591859761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:17.784526117 +0000 UTC m=+2097.403185306" watchObservedRunningTime="2026-10-31 13:38:18.591859761 +0000 UTC m=+2098.210518928"
Oct 31 13:38:18.595668 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:18.592206    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"]
Oct 31 13:38:18.595668 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:38:18.593138    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a52dcc2_c2ab_430c_988e_d3b967f833f3.slice/crio-4265fd12f0f114a9f689c6fce156765802fea8aec3e8223b2c60d8bf85845f6c WatchSource:0}: Error finding container 4265fd12f0f114a9f689c6fce156765802fea8aec3e8223b2c60d8bf85845f6c: Status 404 returned error can't find the container with id 4265fd12f0f114a9f689c6fce156765802fea8aec3e8223b2c60d8bf85845f6c
Oct 31 13:38:18.597824 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:18.597776    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:18.597949 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:18.597903    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:18.598033 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:18.597963    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:18.772406 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:18.772375    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm" event={"ID":"8a52dcc2-c2ab-430c-988e-d3b967f833f3","Type":"ContainerStarted","Data":"e365c0fcce366ead8f5df728c37af26995d99ecaf27ad58f9b5e2ecd7c4644fc"}
Oct 31 13:38:18.772650 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:18.772634    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm" event={"ID":"8a52dcc2-c2ab-430c-988e-d3b967f833f3","Type":"ContainerStarted","Data":"4265fd12f0f114a9f689c6fce156765802fea8aec3e8223b2c60d8bf85845f6c"}
Oct 31 13:38:18.791531 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:18.791505    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:18.798864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:18.798810    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm" podStartSLOduration=1.7987914090000001 podStartE2EDuration="1.798791409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:18.797960178 +0000 UTC m=+2098.416619341" watchObservedRunningTime="2026-10-31 13:38:18.798791409 +0000 UTC m=+2098.417450574"
Oct 31 13:38:18.956581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:18.956536    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:19.782549 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:19.781320    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:19.782549 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:19.782286    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:38:19.973993 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:19.973959    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:20.738774 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:38:20.738731    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:38:20.801177 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:20.801146    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:20.802338 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:20.802258    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:38:20.953712 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:20.953674    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:38:20.954804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:20.954778    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:38:20.965418 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:20.965394    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:38:20.965418 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:20.965409    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:38:20.980017 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:20.979995    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:38:20.980780 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:20.980757    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:38:27.761590 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.761555    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"]
Oct 31 13:38:27.767782 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.767762    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.775442 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.775414    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f4232c-openshift-default-dockercfg-jrtvl\""
Oct 31 13:38:27.781721 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.781690    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"]
Oct 31 13:38:27.854928 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.854889    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.855132 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.855043    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c1be44-f679-4953-9a0d-74f8c160e29c-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.855132 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.855096    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.855132 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.855130    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.855340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.855184    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.855441 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.855421    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt85v\" (UniqueName: \"kubernetes.io/projected/77c1be44-f679-4953-9a0d-74f8c160e29c-kube-api-access-bt85v\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.855505 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.855468    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.855614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.855596    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.855727 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.855709    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.956907 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.956864    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.957092 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.956988    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.957092 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.957027    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.957278 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.957101    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c1be44-f679-4953-9a0d-74f8c160e29c-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.957278 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.957142    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.957278 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.957173    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.957278 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.957212    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.957463 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.957289    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.957463 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.957424    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt85v\" (UniqueName: \"kubernetes.io/projected/77c1be44-f679-4953-9a0d-74f8c160e29c-kube-api-access-bt85v\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.957567 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.957468    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.957567 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.957490    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.957567 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.957549    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.957775 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.957747    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.957905 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.957821    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c1be44-f679-4953-9a0d-74f8c160e29c-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.959396 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.959373    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.959775 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.959752    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.975142 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.975108    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:27.977916 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:27.977886    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt85v\" (UniqueName: \"kubernetes.io/projected/77c1be44-f679-4953-9a0d-74f8c160e29c-kube-api-access-bt85v\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:28.086312 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:28.086220    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:28.452974 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:28.452402    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:28.452974 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:28.452491    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:28.452974 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:28.452533    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:28.470943 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:28.470066    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"]
Oct 31 13:38:28.966319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:28.965978    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk" event={"ID":"77c1be44-f679-4953-9a0d-74f8c160e29c","Type":"ContainerStarted","Data":"3cb8743ac7c7f749484cf0c76811676182f2663853702b79d6b827a01e5448d6"}
Oct 31 13:38:28.966319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:28.966054    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk" event={"ID":"77c1be44-f679-4953-9a0d-74f8c160e29c","Type":"ContainerStarted","Data":"89dea5bdf4d90a88c180c2219d7c44cc20aac77fc6aca764a6998a5def99677e"}
Oct 31 13:38:29.086317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:29.078834    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk" podStartSLOduration=2.062278001 podStartE2EDuration="2.062278001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:29.030144229 +0000 UTC m=+2108.648803395" watchObservedRunningTime="2026-10-31 13:38:29.062278001 +0000 UTC m=+2108.680937159"
Oct 31 13:38:29.101325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:29.092723    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:29.101325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:29.100713    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 13:38:29.101325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:29.100787    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk" podUID="77c1be44-f679-4953-9a0d-74f8c160e29c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 13:38:30.079058 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.078864    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"]
Oct 31 13:38:30.084060 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.084031    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.088491 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.088462    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 13:38:30.088608 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.088514    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk" podUID="77c1be44-f679-4953-9a0d-74f8c160e29c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 13:38:30.108834 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.107741    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-69bdf5\"/\"default-dockercfg-qt8h6\""
Oct 31 13:38:30.108834 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.107838    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-69bdf5\"/\"openshift-service-ca.crt\""
Oct 31 13:38:30.108834 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.107935    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-69bdf5\"/\"kube-root-ca.crt\""
Oct 31 13:38:30.108834 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.108064    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-69bdf5\"/\"test-model-69bdf5-kserve-self-signed-certs\""
Oct 31 13:38:30.185082 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.168039    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"]
Oct 31 13:38:30.185357 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.180636    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982ab639-4caa-445f-b029-4390d81cfad5-tls-certs\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.185593 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.185570    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-home\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.185824 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.185785    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97l7c\" (UniqueName: \"kubernetes.io/projected/982ab639-4caa-445f-b029-4390d81cfad5-kube-api-access-97l7c\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.186095 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.186058    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-dshm\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.186429 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.186395    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-tmp-dir\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.186595 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.186576    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-model-cache\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.315389 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.315351    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982ab639-4caa-445f-b029-4390d81cfad5-tls-certs\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.317685 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.317528    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-home\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.318160 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.318136    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97l7c\" (UniqueName: \"kubernetes.io/projected/982ab639-4caa-445f-b029-4390d81cfad5-kube-api-access-97l7c\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.355110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.323564    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-dshm\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.355110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.323769    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-tmp-dir\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.355110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.323816    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-model-cache\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.355110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.324432    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-model-cache\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.355110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.340647    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-home\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.355110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.340895    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-tmp-dir\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.376720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.362558    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97l7c\" (UniqueName: \"kubernetes.io/projected/982ab639-4caa-445f-b029-4390d81cfad5-kube-api-access-97l7c\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.394338 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.380513    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-dshm\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.395754 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.395718    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982ab639-4caa-445f-b029-4390d81cfad5-tls-certs\") pod \"test-model-69bdf5-kserve-b6566b447-vtv49\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") " pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:30.452369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:30.451447    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:31.071552 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:31.070329    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"]
Oct 31 13:38:31.088066 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:31.088039    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 13:38:31.088500 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:31.088083    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk" podUID="77c1be44-f679-4953-9a0d-74f8c160e29c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 13:38:32.034695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:32.034655    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49" event={"ID":"982ab639-4caa-445f-b029-4390d81cfad5","Type":"ContainerStarted","Data":"9b1f56d1d839b787f5a7549bdc7a876f2fc67b9e2f3355680f5dd94f42dd3bd5"}
Oct 31 13:38:32.034979 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:32.034951    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49" event={"ID":"982ab639-4caa-445f-b029-4390d81cfad5","Type":"ContainerStarted","Data":"e457914ac333e0bd7b9efc3d97c4a012ca51357a4bdabff170e49fca039cd1e7"}
Oct 31 13:38:32.090373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:32.090338    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 13:38:32.091128 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:32.091096    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk" podUID="77c1be44-f679-4953-9a0d-74f8c160e29c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 13:38:32.105900 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:32.105845    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49" podStartSLOduration=2.10582513 podStartE2EDuration="2.10582513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:32.071190622 +0000 UTC m=+2111.689849786" watchObservedRunningTime="2026-10-31 13:38:32.10582513 +0000 UTC m=+2111.724484298"
Oct 31 13:38:32.458320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:32.456394    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:33.087445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:33.087409    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 13:38:33.087615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:33.087460    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk" podUID="77c1be44-f679-4953-9a0d-74f8c160e29c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 13:38:34.087124 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:34.087087    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 13:38:34.087608 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:34.087151    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk" podUID="77c1be44-f679-4953-9a0d-74f8c160e29c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 13:38:35.086891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:35.086863    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 13:38:35.087104 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:35.087077    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk" podUID="77c1be44-f679-4953-9a0d-74f8c160e29c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 13:38:36.092211 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:36.092179    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:36.092691 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:36.092668    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:36.093413 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:36.093394    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:38:40.462332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:40.461349    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:40.573442 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:40.573402    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:41.466280 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:41.466253    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:38:50.743682 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:38:50.743623    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:38:51.480542 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:51.480499    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"]
Oct 31 13:38:51.481623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:51.481383    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh" podUID="30b88da6-a8a1-4882-8264-3ae18b1350a9" containerName="istio-proxy" containerID="cri-o://41b0a19464ccfc17222560df567a09d33c0801873744b12491be9bd0ca908c59" gracePeriod=30
Oct 31 13:38:51.946782 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:51.946734    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"]
Oct 31 13:38:51.991908 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:51.991874    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"]
Oct 31 13:38:51.992107 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:51.991995    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:51.997156 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:51.997074    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d4b52a\"/\"kube-root-ca.crt\""
Oct 31 13:38:51.997470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:51.997439    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d4b52a\"/\"openshift-service-ca.crt\""
Oct 31 13:38:51.999190 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:51.999072    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d4b52a\"/\"default-dockercfg-qvdmx\""
Oct 31 13:38:51.999190 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:51.999168    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d4b52a\"/\"test-model-d4b52a-kserve-self-signed-certs\""
Oct 31 13:38:52.200385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.188779    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-model-cache\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.200385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.188859    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-home\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.200385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.188885    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-tmp-dir\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.200385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.188999    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-dshm\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.200385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.189094    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e003dfad-238c-47f3-aa19-696ca2c1264a-tls-certs\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.200385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.189177    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgs9k\" (UniqueName: \"kubernetes.io/projected/e003dfad-238c-47f3-aa19-696ca2c1264a-kube-api-access-fgs9k\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.294109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.293724    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-model-cache\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.294109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.293803    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-home\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.294109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.293832    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-tmp-dir\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.294109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.293950    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-dshm\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.294109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.294052    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e003dfad-238c-47f3-aa19-696ca2c1264a-tls-certs\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.294552 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.294137    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgs9k\" (UniqueName: \"kubernetes.io/projected/e003dfad-238c-47f3-aa19-696ca2c1264a-kube-api-access-fgs9k\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.294552 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.294456    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-model-cache\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.294733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.294711    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-home\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.295551 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.295439    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-tmp-dir\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.318320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.315972    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e003dfad-238c-47f3-aa19-696ca2c1264a-tls-certs\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.331355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.329391    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-dshm\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.334223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.334049    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgs9k\" (UniqueName: \"kubernetes.io/projected/e003dfad-238c-47f3-aa19-696ca2c1264a-kube-api-access-fgs9k\") pod \"test-model-d4b52a-kserve-9cd9fb59-r2ck2\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") " pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:52.619587 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:52.618486    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:53.235286 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.235250    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"]
Oct 31 13:38:53.507693 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.507528    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"]
Oct 31 13:38:53.518311 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.518270    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.529880 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.525397    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-554759-openshift-default-dockercfg-bqqlh\""
Oct 31 13:38:53.539630 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.539604    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"]
Oct 31 13:38:53.572349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.570397    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-envoy\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.572349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.570518    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpm9d\" (UniqueName: \"kubernetes.io/projected/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-kube-api-access-fpm9d\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.572349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.570561    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-data\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.572349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.570617    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-podinfo\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.572349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.570690    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-token\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.572349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.571136    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-credential-socket\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.572349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.571249    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-workload-certs\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.572349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.571338    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istiod-ca-cert\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.572349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.571368    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-workload-socket\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.672585 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.672508    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-podinfo\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.672585 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.672585    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-token\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.672829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.672628    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-credential-socket\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.672829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.672739    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-workload-certs\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.672829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.672805    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istiod-ca-cert\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.672990 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.672832    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-workload-socket\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.672990 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.672984    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-envoy\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.673137 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.673107    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpm9d\" (UniqueName: \"kubernetes.io/projected/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-kube-api-access-fpm9d\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.673196 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.673145    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-data\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.673251 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.673201    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-workload-certs\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.673856 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.673833    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-credential-socket\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.674009 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.673872    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-workload-socket\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.674115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.674008    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istiod-ca-cert\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.674201 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.674110    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-data\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.680024 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.675594    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-podinfo\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.710931 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.698763    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-envoy\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.710931 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.705739    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpm9d\" (UniqueName: \"kubernetes.io/projected/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-kube-api-access-fpm9d\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.713852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.713814    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-token\") pod \"e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") " pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.851558 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.849601    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:53.896909 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.896878    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2" event={"ID":"e003dfad-238c-47f3-aa19-696ca2c1264a","Type":"ContainerStarted","Data":"eb36704f525a37940eda180b3d3a81318f6c221a5daf2c5510d3509290bd2284"}
Oct 31 13:38:53.897139 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:53.897118    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2" event={"ID":"e003dfad-238c-47f3-aa19-696ca2c1264a","Type":"ContainerStarted","Data":"8e9dc81a25f759cdc5a7f1d0088c700f18e8daf2737ce50c4f44d50579e57164"}
Oct 31 13:38:54.214277 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:38:54.214243    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd757f7_0d36_4d30_a6ab_94e8f1b2cc9e.slice/crio-de229071f12a6594355a225e629d43eb889234e7f6bd90bfebc99bfe502e8aeb WatchSource:0}: Error finding container de229071f12a6594355a225e629d43eb889234e7f6bd90bfebc99bfe502e8aeb: Status 404 returned error can't find the container with id de229071f12a6594355a225e629d43eb889234e7f6bd90bfebc99bfe502e8aeb
Oct 31 13:38:54.217449 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:54.217389    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2" podStartSLOduration=3.217371423 podStartE2EDuration="3.217371423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:53.933435253 +0000 UTC m=+2133.552094418" watchObservedRunningTime="2026-10-31 13:38:54.217371423 +0000 UTC m=+2133.836030588"
Oct 31 13:38:54.217600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:54.217581    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"]
Oct 31 13:38:54.218218 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:54.218176    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:54.218329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:54.218263    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:54.218401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:54.218331    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:54.619638 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:54.619548    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:38:54.949360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:54.948914    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq" event={"ID":"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e","Type":"ContainerStarted","Data":"59552b38c6f5b1cb11d559a1d59ba6b0dccdec5d8af4c337c73d3b98417a15c4"}
Oct 31 13:38:54.949360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:54.948972    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq" event={"ID":"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e","Type":"ContainerStarted","Data":"de229071f12a6594355a225e629d43eb889234e7f6bd90bfebc99bfe502e8aeb"}
Oct 31 13:38:55.851012 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:55.850980    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:55.857645 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:55.857615    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:55.901202 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:55.901145    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq" podStartSLOduration=2.901129228 podStartE2EDuration="2.901129228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:54.997533892 +0000 UTC m=+2134.616193058" watchObservedRunningTime="2026-10-31 13:38:55.901129228 +0000 UTC m=+2135.519788392"
Oct 31 13:38:55.973648 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:55.973602    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:55.975355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:55.975331    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:38:56.367042 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.367006    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"]
Oct 31 13:38:56.372183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.372157    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.379548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.379519    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-61da9d-openshift-default-dockercfg-nzt6j\""
Oct 31 13:38:56.383636 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.383606    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"]
Oct 31 13:38:56.482307 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.482256    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.482470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.482426    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjp79\" (UniqueName: \"kubernetes.io/projected/239235d5-45ef-405c-8c3a-02618f5ebaa0-kube-api-access-rjp79\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.482563 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.482525    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.482610 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.482602    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.482801 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.482780    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.482880 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.482824    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.482951 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.482928    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239235d5-45ef-405c-8c3a-02618f5ebaa0-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.483102 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.483084    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.483189 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.483134    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.585076 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.585048    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjp79\" (UniqueName: \"kubernetes.io/projected/239235d5-45ef-405c-8c3a-02618f5ebaa0-kube-api-access-rjp79\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.585237 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.585115    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.585237 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.585169    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.585411 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.585380    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.585469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.585423    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.585524 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.585478    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239235d5-45ef-405c-8c3a-02618f5ebaa0-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.585577 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.585524    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.585686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.585665    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.585763 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.585681    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.585763 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.585741    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.585876 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.585767    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.585876 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.585851    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.586023 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.586001    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.586237 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.586219    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239235d5-45ef-405c-8c3a-02618f5ebaa0-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.588012 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.587992    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.588234 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.588214    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.603442 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.603398    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjp79\" (UniqueName: \"kubernetes.io/projected/239235d5-45ef-405c-8c3a-02618f5ebaa0-kube-api-access-rjp79\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.603733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.603712    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.734787 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.716488    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:56.903506 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:56.903469    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:38:57.030072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.021220    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-credential-socket\") pod \"30b88da6-a8a1-4882-8264-3ae18b1350a9\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") "
Oct 31 13:38:57.030072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.021321    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b88da6-a8a1-4882-8264-3ae18b1350a9-kube-api-access-s4mj4\" (UniqueName: \"kubernetes.io/projected/30b88da6-a8a1-4882-8264-3ae18b1350a9-kube-api-access-s4mj4\") pod \"30b88da6-a8a1-4882-8264-3ae18b1350a9\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") "
Oct 31 13:38:57.030072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.021384    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-workload-socket\") pod \"30b88da6-a8a1-4882-8264-3ae18b1350a9\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") "
Oct 31 13:38:57.030072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.021572    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-envoy\") pod \"30b88da6-a8a1-4882-8264-3ae18b1350a9\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") "
Oct 31 13:38:57.030072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.021623    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-workload-certs\") pod \"30b88da6-a8a1-4882-8264-3ae18b1350a9\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") "
Oct 31 13:38:57.030072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.021667    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-token\" (UniqueName: \"kubernetes.io/projected/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-token\") pod \"30b88da6-a8a1-4882-8264-3ae18b1350a9\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") "
Oct 31 13:38:57.030072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.021770    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-data\") pod \"30b88da6-a8a1-4882-8264-3ae18b1350a9\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") "
Oct 31 13:38:57.030072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.021810    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-podinfo\") pod \"30b88da6-a8a1-4882-8264-3ae18b1350a9\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") "
Oct 31 13:38:57.030072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.022010    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b88da6-a8a1-4882-8264-3ae18b1350a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b88da6-a8a1-4882-8264-3ae18b1350a9-istiod-ca-cert\") pod \"30b88da6-a8a1-4882-8264-3ae18b1350a9\" (UID: \"30b88da6-a8a1-4882-8264-3ae18b1350a9\") "
Oct 31 13:38:57.030072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.023388    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b88da6-a8a1-4882-8264-3ae18b1350a9-istiod-ca-cert" pod "30b88da6-a8a1-4882-8264-3ae18b1350a9" (UID: "30b88da6-a8a1-4882-8264-3ae18b1350a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:38:57.030072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.029940    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-credential-socket" pod "30b88da6-a8a1-4882-8264-3ae18b1350a9" (UID: "30b88da6-a8a1-4882-8264-3ae18b1350a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:38:57.030802 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.030573    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-workload-certs" pod "30b88da6-a8a1-4882-8264-3ae18b1350a9" (UID: "30b88da6-a8a1-4882-8264-3ae18b1350a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:38:57.030802 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.030757    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-workload-socket" pod "30b88da6-a8a1-4882-8264-3ae18b1350a9" (UID: "30b88da6-a8a1-4882-8264-3ae18b1350a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:38:57.047394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.038403    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-data" pod "30b88da6-a8a1-4882-8264-3ae18b1350a9" (UID: "30b88da6-a8a1-4882-8264-3ae18b1350a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:38:57.056374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.053224    4660 generic.go:356] "Generic (PLEG): container finished" podID="30b88da6-a8a1-4882-8264-3ae18b1350a9" containerID="41b0a19464ccfc17222560df567a09d33c0801873744b12491be9bd0ca908c59" exitCode=0
Oct 31 13:38:57.056374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.054554    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"
Oct 31 13:38:57.056374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.055004    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh" event={"ID":"30b88da6-a8a1-4882-8264-3ae18b1350a9","Type":"ContainerDied","Data":"41b0a19464ccfc17222560df567a09d33c0801873744b12491be9bd0ca908c59"}
Oct 31 13:38:57.056374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.055054    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh" event={"ID":"30b88da6-a8a1-4882-8264-3ae18b1350a9","Type":"ContainerDied","Data":"4e855d0acd8ba05065b86948473b0018c8b36ad9553bf8ee75f9636a12d5e19f"}
Oct 31 13:38:57.056374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.055083    4660 scope.go:122] "RemoveContainer" containerID="41b0a19464ccfc17222560df567a09d33c0801873744b12491be9bd0ca908c59"
Oct 31 13:38:57.085622 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.085130    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-envoy" pod "30b88da6-a8a1-4882-8264-3ae18b1350a9" (UID: "30b88da6-a8a1-4882-8264-3ae18b1350a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:38:57.085622 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.085130    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-podinfo" pod "30b88da6-a8a1-4882-8264-3ae18b1350a9" (UID: "30b88da6-a8a1-4882-8264-3ae18b1350a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:38:57.086493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.086451    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b88da6-a8a1-4882-8264-3ae18b1350a9-kube-api-access-s4mj4" pod "30b88da6-a8a1-4882-8264-3ae18b1350a9" (UID: "30b88da6-a8a1-4882-8264-3ae18b1350a9"). InnerVolumeSpecName "kube-api-access-s4mj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:38:57.089128 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.089085    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-token" pod "30b88da6-a8a1-4882-8264-3ae18b1350a9" (UID: "30b88da6-a8a1-4882-8264-3ae18b1350a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:38:57.105326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.105272    4660 scope.go:122] "RemoveContainer" containerID="ad3f9d5af33a976432799429a354c069336a507eb64096de4eda3117c922e778"
Oct 31 13:38:57.118563 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.118210    4660 scope.go:122] "RemoveContainer" containerID="41b0a19464ccfc17222560df567a09d33c0801873744b12491be9bd0ca908c59"
Oct 31 13:38:57.118563 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:38:57.118547    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b0a19464ccfc17222560df567a09d33c0801873744b12491be9bd0ca908c59\": container with ID starting with 41b0a19464ccfc17222560df567a09d33c0801873744b12491be9bd0ca908c59 not found: ID does not exist" containerID="41b0a19464ccfc17222560df567a09d33c0801873744b12491be9bd0ca908c59"
Oct 31 13:38:57.118697 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.118586    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b0a19464ccfc17222560df567a09d33c0801873744b12491be9bd0ca908c59"} err="failed to get container status \"41b0a19464ccfc17222560df567a09d33c0801873744b12491be9bd0ca908c59\": rpc error: code = NotFound desc = could not find container \"41b0a19464ccfc17222560df567a09d33c0801873744b12491be9bd0ca908c59\": container with ID starting with 41b0a19464ccfc17222560df567a09d33c0801873744b12491be9bd0ca908c59 not found: ID does not exist"
Oct 31 13:38:57.118697 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.118624    4660 scope.go:122] "RemoveContainer" containerID="ad3f9d5af33a976432799429a354c069336a507eb64096de4eda3117c922e778"
Oct 31 13:38:57.119020 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:38:57.118995    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3f9d5af33a976432799429a354c069336a507eb64096de4eda3117c922e778\": container with ID starting with ad3f9d5af33a976432799429a354c069336a507eb64096de4eda3117c922e778 not found: ID does not exist" containerID="ad3f9d5af33a976432799429a354c069336a507eb64096de4eda3117c922e778"
Oct 31 13:38:57.119061 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.119030    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3f9d5af33a976432799429a354c069336a507eb64096de4eda3117c922e778"} err="failed to get container status \"ad3f9d5af33a976432799429a354c069336a507eb64096de4eda3117c922e778\": rpc error: code = NotFound desc = could not find container \"ad3f9d5af33a976432799429a354c069336a507eb64096de4eda3117c922e778\": container with ID starting with ad3f9d5af33a976432799429a354c069336a507eb64096de4eda3117c922e778 not found: ID does not exist"
Oct 31 13:38:57.124636 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.124613    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b88da6-a8a1-4882-8264-3ae18b1350a9-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:38:57.124636 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.124636    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:38:57.124780 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.124646    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4mj4\" (UniqueName: \"kubernetes.io/projected/30b88da6-a8a1-4882-8264-3ae18b1350a9-kube-api-access-s4mj4\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:38:57.124780 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.124656    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:38:57.124780 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.124664    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:38:57.124780 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.124672    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:38:57.124780 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.124680    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:38:57.124780 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.124699    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:38:57.124780 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.124707    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b88da6-a8a1-4882-8264-3ae18b1350a9-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:38:57.135640 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.135616    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"]
Oct 31 13:38:57.137237 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:38:57.137213    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239235d5_45ef_405c_8c3a_02618f5ebaa0.slice/crio-8ca7c9c50b911a9d275cd847eb3f44c39d11da72995696adbc15c6a25aaba25d WatchSource:0}: Error finding container 8ca7c9c50b911a9d275cd847eb3f44c39d11da72995696adbc15c6a25aaba25d: Status 404 returned error can't find the container with id 8ca7c9c50b911a9d275cd847eb3f44c39d11da72995696adbc15c6a25aaba25d
Oct 31 13:38:57.140367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.140334    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:57.140451 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.140403    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:57.140451 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.140434    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:57.395137 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.395102    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"]
Oct 31 13:38:57.401250 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.401222    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c45a37-openshift-default-84b65d78b5-4cfhh"]
Oct 31 13:38:57.868956 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.868922    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"]
Oct 31 13:38:57.869686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.869668    4660 cpu_manager.go:414] "containerMap: removing container" podUID="30b88da6-a8a1-4882-8264-3ae18b1350a9" containerName="istio-proxy"
Oct 31 13:38:57.869769 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.869714    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b88da6-a8a1-4882-8264-3ae18b1350a9" containerName="istio-proxy"
Oct 31 13:38:57.869908 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.869897    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b88da6-a8a1-4882-8264-3ae18b1350a9" containerName="istio-proxy"
Oct 31 13:38:57.869953 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.869914    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b88da6-a8a1-4882-8264-3ae18b1350a9" containerName="istio-proxy"
Oct 31 13:38:57.873999 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.873974    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:57.891513 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.891487    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"]
Oct 31 13:38:57.892651 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.892622    4660 cpu_manager.go:414] "containerMap: removing container" podUID="30b88da6-a8a1-4882-8264-3ae18b1350a9" containerName="istio-proxy"
Oct 31 13:38:57.892651 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.892649    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b88da6-a8a1-4882-8264-3ae18b1350a9" containerName="istio-proxy"
Oct 31 13:38:57.897271 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.897239    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:57.898200 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.898176    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"]
Oct 31 13:38:57.948318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.941378    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"]
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960175    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960237    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960276    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960371    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960429    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960455    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960478    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960520    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960553    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkllf\" (UniqueName: \"kubernetes.io/projected/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-kube-api-access-lkllf\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960603    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4a9639-2171-4539-abca-aa29d66bcdfb-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960640    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960681    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960805    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960856    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccmpx\" (UniqueName: \"kubernetes.io/projected/0e4a9639-2171-4539-abca-aa29d66bcdfb-kube-api-access-ccmpx\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960879    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960905    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.960953    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:57.961215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:57.961019    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.062389 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.062349    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.062582 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.062440    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.062582 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.062483    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.062582 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.062550    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.062784 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.062604    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.062784 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.062688    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.062784 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.062748    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.062784 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.062779    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.062994 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.062807    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.062994 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.062850    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.062994 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.062880    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkllf\" (UniqueName: \"kubernetes.io/projected/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-kube-api-access-lkllf\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.062994 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.062894    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.062994 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.062945    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4a9639-2171-4539-abca-aa29d66bcdfb-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.062994 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.062955    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.062994 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.062975    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.063359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.062999    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.063359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.063013    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.063359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.063166    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.063359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.063190    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.063359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.063268    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccmpx\" (UniqueName: \"kubernetes.io/projected/0e4a9639-2171-4539-abca-aa29d66bcdfb-kube-api-access-ccmpx\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.063359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.063341    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.063667 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.063383    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.063667 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.063501    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.063854 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.063808    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.064098 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.063934    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.064098 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.064058    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.064504 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.064464    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.065020 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.064996    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4a9639-2171-4539-abca-aa29d66bcdfb-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.066002 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.065975    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.067069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.066403    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.067069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.066550    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.067069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.066663    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.067069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.066934    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj" event={"ID":"239235d5-45ef-405c-8c3a-02618f5ebaa0","Type":"ContainerStarted","Data":"dca0d9584297e37f794a7abb2dcd6cab6829b6f751396b5a81d2c2ed44b534e9"}
Oct 31 13:38:58.067069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.066978    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj" event={"ID":"239235d5-45ef-405c-8c3a-02618f5ebaa0","Type":"ContainerStarted","Data":"8ca7c9c50b911a9d275cd847eb3f44c39d11da72995696adbc15c6a25aaba25d"}
Oct 31 13:38:58.076710 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.076688    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.076881 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.076812    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkllf\" (UniqueName: \"kubernetes.io/projected/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-kube-api-access-lkllf\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.077353 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.077334    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccmpx\" (UniqueName: \"kubernetes.io/projected/0e4a9639-2171-4539-abca-aa29d66bcdfb-kube-api-access-ccmpx\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.077587 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.077569    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.099527 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.099483    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj" podStartSLOduration=2.099469386 podStartE2EDuration="2.099469386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:58.093989859 +0000 UTC m=+2137.712649025" watchObservedRunningTime="2026-10-31 13:38:58.099469386 +0000 UTC m=+2137.718128549"
Oct 31 13:38:58.211321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.198638    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:38:58.247324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.246615    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:38:58.579002 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.578961    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b88da6-a8a1-4882-8264-3ae18b1350a9" path="/var/lib/kubelet/pods/30b88da6-a8a1-4882-8264-3ae18b1350a9/volumes"
Oct 31 13:38:58.730730 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.730694    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:38:58.734537 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.734507    4660 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 31 13:38:58.734750 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.734726    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj" podUID="239235d5-45ef-405c-8c3a-02618f5ebaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 31 13:38:58.893329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.893286    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"]
Oct 31 13:38:58.900667 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:38:58.900470    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef53539_9688_4af7_8ce0_5aa0bbea4f9c.slice/crio-c063f362ef1bb99dfaae8bdba72ba61731d19a6ce94ba411b5a11f96769596a1 WatchSource:0}: Error finding container c063f362ef1bb99dfaae8bdba72ba61731d19a6ce94ba411b5a11f96769596a1: Status 404 returned error can't find the container with id c063f362ef1bb99dfaae8bdba72ba61731d19a6ce94ba411b5a11f96769596a1
Oct 31 13:38:58.908614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.907934    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:58.908614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.908022    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:58.908614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.908089    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:58.977887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.951153    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:58.977887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.951264    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:58.977887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:58.951330    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:38:59.073314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:59.067354    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"]
Oct 31 13:38:59.273324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:59.267455    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92" event={"ID":"0e4a9639-2171-4539-abca-aa29d66bcdfb","Type":"ContainerStarted","Data":"195e928d1c705b184b34c73e9c5ae1918cfcb00128594503c767f896ff1b044c"}
Oct 31 13:38:59.288337 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:59.276263    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p" event={"ID":"fef53539-9688-4af7-8ce0-5aa0bbea4f9c","Type":"ContainerStarted","Data":"c063f362ef1bb99dfaae8bdba72ba61731d19a6ce94ba411b5a11f96769596a1"}
Oct 31 13:38:59.726440 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:59.725923    4660 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 31 13:38:59.726440 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:38:59.725983    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj" podUID="239235d5-45ef-405c-8c3a-02618f5ebaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 31 13:39:00.286027 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:00.285983    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92" event={"ID":"0e4a9639-2171-4539-abca-aa29d66bcdfb","Type":"ContainerStarted","Data":"d72aed58664859312761d9239e80d55c974c2f4e2bc54a6ef958b1e9fa3df567"}
Oct 31 13:39:00.290240 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:00.290203    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p" event={"ID":"fef53539-9688-4af7-8ce0-5aa0bbea4f9c","Type":"ContainerStarted","Data":"83a6a07a9fb726bf2c9b09bf6a1ff33cf38a7e08bf0ebfc423327f3cb66a9036"}
Oct 31 13:39:00.347358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:00.343708    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92" podStartSLOduration=3.343687317 podStartE2EDuration="3.343687317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:00.317803136 +0000 UTC m=+2139.936462314" watchObservedRunningTime="2026-10-31 13:39:00.343687317 +0000 UTC m=+2139.962346483"
Oct 31 13:39:00.351473 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:00.351425    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p" podStartSLOduration=3.351413372 podStartE2EDuration="3.351413372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:00.344858026 +0000 UTC m=+2139.963517192" watchObservedRunningTime="2026-10-31 13:39:00.351413372 +0000 UTC m=+2139.970072535"
Oct 31 13:39:00.732320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:00.731881    4660 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 31 13:39:00.732320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:00.731943    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj" podUID="239235d5-45ef-405c-8c3a-02618f5ebaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 31 13:39:01.310317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:01.307853    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:39:01.310317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:01.307921    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:39:01.395861 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:01.395822    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 13:39:01.396160 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:01.396124    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p" podUID="fef53539-9688-4af7-8ce0-5aa0bbea4f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 13:39:01.442941 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:01.442905    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 31 13:39:01.443459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:01.443427    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92" podUID="0e4a9639-2171-4539-abca-aa29d66bcdfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 31 13:39:01.817387 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:01.817349    4660 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 31 13:39:01.817581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:01.817421    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj" podUID="239235d5-45ef-405c-8c3a-02618f5ebaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 31 13:39:02.200453 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:02.200414    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 31 13:39:02.200768 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:02.200720    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92" podUID="0e4a9639-2171-4539-abca-aa29d66bcdfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 31 13:39:02.251319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:02.249719    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 13:39:02.251319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:02.249777    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p" podUID="fef53539-9688-4af7-8ce0-5aa0bbea4f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 13:39:02.314320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:02.311393    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:39:02.726570 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:02.726538    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:39:02.742074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:02.741594    4660 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 31 13:39:02.742074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:02.741653    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj" podUID="239235d5-45ef-405c-8c3a-02618f5ebaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 31 13:39:03.100440 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:03.100122    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:39:03.200369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:03.200273    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 31 13:39:03.200618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:03.200582    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92" podUID="0e4a9639-2171-4539-abca-aa29d66bcdfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 31 13:39:03.249354 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:03.249319    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 13:39:03.249604 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:03.249571    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p" podUID="fef53539-9688-4af7-8ce0-5aa0bbea4f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 13:39:03.729320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:03.726671    4660 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 31 13:39:03.729320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:03.726772    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj" podUID="239235d5-45ef-405c-8c3a-02618f5ebaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 31 13:39:03.740321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:03.740002    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:39:04.245318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:04.230713    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 31 13:39:04.245318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:04.230777    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92" podUID="0e4a9639-2171-4539-abca-aa29d66bcdfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 31 13:39:04.295316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:04.295156    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 13:39:04.295316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:04.295231    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p" podUID="fef53539-9688-4af7-8ce0-5aa0bbea4f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 13:39:04.925675 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:04.925641    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:39:04.955316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:04.927914    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:39:04.955609 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:04.955587    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:39:05.210369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:05.201676    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 31 13:39:05.210369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:05.201766    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92" podUID="0e4a9639-2171-4539-abca-aa29d66bcdfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 31 13:39:05.252316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:05.251679    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 13:39:05.252316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:05.251736    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p" podUID="fef53539-9688-4af7-8ce0-5aa0bbea4f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 13:39:06.240277 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:06.240236    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 31 13:39:06.242478 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:06.242443    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92" podUID="0e4a9639-2171-4539-abca-aa29d66bcdfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 31 13:39:06.251808 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:06.251770    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 13:39:06.251965 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:06.251826    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p" podUID="fef53539-9688-4af7-8ce0-5aa0bbea4f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 13:39:07.205719 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:07.205677    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:39:07.206199 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:07.206187    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:39:07.207082 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:07.207059    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:39:07.253081 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:07.253048    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:39:07.253851 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:07.253478    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:39:07.254452 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:07.254433    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:39:16.253376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:16.253338    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"]
Oct 31 13:39:16.282351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:16.282318    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:39:16.291442 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:16.290123    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0d835b-serving-cert\""
Oct 31 13:39:16.296955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:16.296738    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"]
Oct 31 13:39:16.593317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:16.586408    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndmjj\" (UniqueName: \"kubernetes.io/projected/51ef7324-3450-4432-a5ee-30c91193f7a0-kube-api-access-ndmjj\") pod \"maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h\" (UID: \"51ef7324-3450-4432-a5ee-30c91193f7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:39:16.593317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:16.586510    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51ef7324-3450-4432-a5ee-30c91193f7a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h\" (UID: \"51ef7324-3450-4432-a5ee-30c91193f7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:39:16.593317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:16.586565    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51ef7324-3450-4432-a5ee-30c91193f7a0-metrics-certs\") pod \"maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h\" (UID: \"51ef7324-3450-4432-a5ee-30c91193f7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:39:16.593317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:16.586766    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ef7324-3450-4432-a5ee-30c91193f7a0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h\" (UID: \"51ef7324-3450-4432-a5ee-30c91193f7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:39:16.709254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:16.701596    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51ef7324-3450-4432-a5ee-30c91193f7a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h\" (UID: \"51ef7324-3450-4432-a5ee-30c91193f7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:39:16.709254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:16.701685    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51ef7324-3450-4432-a5ee-30c91193f7a0-metrics-certs\") pod \"maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h\" (UID: \"51ef7324-3450-4432-a5ee-30c91193f7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:39:16.709254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:16.704535    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ef7324-3450-4432-a5ee-30c91193f7a0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h\" (UID: \"51ef7324-3450-4432-a5ee-30c91193f7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:39:16.709254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:16.707680    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ef7324-3450-4432-a5ee-30c91193f7a0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h\" (UID: \"51ef7324-3450-4432-a5ee-30c91193f7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:39:16.709734 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:16.709444    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndmjj\" (UniqueName: \"kubernetes.io/projected/51ef7324-3450-4432-a5ee-30c91193f7a0-kube-api-access-ndmjj\") pod \"maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h\" (UID: \"51ef7324-3450-4432-a5ee-30c91193f7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:39:16.811391 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:16.804561    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51ef7324-3450-4432-a5ee-30c91193f7a0-metrics-certs\") pod \"maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h\" (UID: \"51ef7324-3450-4432-a5ee-30c91193f7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:39:16.821941 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:16.819185    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51ef7324-3450-4432-a5ee-30c91193f7a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h\" (UID: \"51ef7324-3450-4432-a5ee-30c91193f7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:39:16.848991 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:16.828954    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndmjj\" (UniqueName: \"kubernetes.io/projected/51ef7324-3450-4432-a5ee-30c91193f7a0-kube-api-access-ndmjj\") pod \"maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h\" (UID: \"51ef7324-3450-4432-a5ee-30c91193f7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:39:17.011565 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.011529    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:39:17.377366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.375397    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"]
Oct 31 13:39:17.489928 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.489895    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"]
Oct 31 13:39:17.496203 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.496178    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.524196 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.522908    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"]
Oct 31 13:39:17.531200 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.530056    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f4232c\"/\"default-dockercfg-qr8lp\""
Oct 31 13:39:17.531200 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.530211    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f4232c\"/\"test-model-f4232c-kserve-self-signed-certs\""
Oct 31 13:39:17.559306 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.559266    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc22df5-69c4-4def-9e7d-f022057cd410-tls-certs\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.559489 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.559357    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-home\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.559489 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.559475    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-model-cache\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.559633 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.559612    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spjlf\" (UniqueName: \"kubernetes.io/projected/cdc22df5-69c4-4def-9e7d-f022057cd410-kube-api-access-spjlf\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.559690 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.559662    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-dshm\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.559866 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.559849    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-tmp-dir\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.593699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.593655    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f4232c\"/\"kube-root-ca.crt\""
Oct 31 13:39:17.593951 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.593898    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f4232c\"/\"openshift-service-ca.crt\""
Oct 31 13:39:17.670526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.669277    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spjlf\" (UniqueName: \"kubernetes.io/projected/cdc22df5-69c4-4def-9e7d-f022057cd410-kube-api-access-spjlf\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.670526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.669369    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-dshm\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.670526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.669592    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-tmp-dir\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.670526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.669763    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc22df5-69c4-4def-9e7d-f022057cd410-tls-certs\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.670526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.669792    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-home\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.670526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.669836    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-model-cache\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.670526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.670057    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-tmp-dir\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.670526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.670373    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-model-cache\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.670526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.670392    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-home\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.672464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.672431    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc22df5-69c4-4def-9e7d-f022057cd410-tls-certs\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.672607 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.672581    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-dshm\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.696541 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.696507    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spjlf\" (UniqueName: \"kubernetes.io/projected/cdc22df5-69c4-4def-9e7d-f022057cd410-kube-api-access-spjlf\") pod \"test-model-f4232c-kserve-667644466c-fm7pm\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") " pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:17.823521 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:17.823485    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:18.182893 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:18.175534    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h" event={"ID":"51ef7324-3450-4432-a5ee-30c91193f7a0","Type":"ContainerStarted","Data":"afff2fc6505e13e9541b0be8bdaaa1b88a80050e2ed440ce756915b8f38b8107"}
Oct 31 13:39:18.540362 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:18.540159    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"]
Oct 31 13:39:19.187560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.187520    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm" event={"ID":"cdc22df5-69c4-4def-9e7d-f022057cd410","Type":"ContainerStarted","Data":"e2e232f7db8a59e15bdbeee788152a38f524689440cd17bf45dd72c5922963c4"}
Oct 31 13:39:19.187763 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.187573    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm" event={"ID":"cdc22df5-69c4-4def-9e7d-f022057cd410","Type":"ContainerStarted","Data":"8f4d0285350ca6b165583a22f8b1262e396a1d2794685235cc47cd23aaca51a9"}
Oct 31 13:39:19.189503 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.189473    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h" event={"ID":"51ef7324-3450-4432-a5ee-30c91193f7a0","Type":"ContainerStarted","Data":"b66941891f46556271083fc751020a3063f974ce356407c9f01e8942d1ff4c5c"}
Oct 31 13:39:19.189635 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.189601    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:39:19.210730 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.210681    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm" podStartSLOduration=2.210663954 podStartE2EDuration="2.210663954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:19.209021792 +0000 UTC m=+2158.827680958" watchObservedRunningTime="2026-10-31 13:39:19.210663954 +0000 UTC m=+2158.829323119"
Oct 31 13:39:19.240769 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.240705    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h" podStartSLOduration=2.791475916 podStartE2EDuration="3.240685873s" totalImagesPullingTime="449.209957ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:19.230065746 +0000 UTC m=+2158.848724901" watchObservedRunningTime="2026-10-31 13:39:19.240685873 +0000 UTC m=+2158.859345041"
Oct 31 13:39:19.679523 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.679489    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"]
Oct 31 13:39:19.683951 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.683924    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:39:19.697076 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.696468    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1ddade-serving-cert\""
Oct 31 13:39:19.702922 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.702087    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"]
Oct 31 13:39:19.764153 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.764121    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-metrics-certs\") pod \"maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv\" (UID: \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:39:19.764356 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.764278    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv\" (UID: \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:39:19.764432 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.764359    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chbzp\" (UniqueName: \"kubernetes.io/projected/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-kube-api-access-chbzp\") pod \"maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv\" (UID: \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:39:19.764495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.764474    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-maas-api-tls\") pod \"maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv\" (UID: \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:39:19.845610 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.845417    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:19.866366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.866288    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-metrics-certs\") pod \"maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv\" (UID: \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:39:19.866584 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.866509    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv\" (UID: \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:39:19.866584 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.866561    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chbzp\" (UniqueName: \"kubernetes.io/projected/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-kube-api-access-chbzp\") pod \"maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv\" (UID: \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:39:19.866712 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.866697    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-maas-api-tls\") pod \"maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv\" (UID: \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:39:19.867347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.867318    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv\" (UID: \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:39:19.869113 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.869085    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-metrics-certs\") pod \"maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv\" (UID: \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:39:19.869408 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.869385    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-maas-api-tls\") pod \"maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv\" (UID: \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:39:19.879858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:19.879835    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chbzp\" (UniqueName: \"kubernetes.io/projected/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-kube-api-access-chbzp\") pod \"maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv\" (UID: \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:39:20.022801 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:20.022702    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:39:20.579780 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:20.579750    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"]
Oct 31 13:39:20.748331 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:39:20.748247    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:39:21.212550 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:21.212509    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv" event={"ID":"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c","Type":"ContainerStarted","Data":"724444f5e3bcf6a7025378e5075d391c707c1dcdb5d75a1ab4a67257960932ac"}
Oct 31 13:39:21.212725 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:21.212577    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv" event={"ID":"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c","Type":"ContainerStarted","Data":"7e5a5620b3c05db741f6f4642b16d707604ef63df90dcffeb5fc7e7f7a9d665c"}
Oct 31 13:39:21.212725 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:21.212607    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:39:21.241832 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:21.241779    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv" podStartSLOduration=1.844886175 podStartE2EDuration="2.241763821s" totalImagesPullingTime="396.877646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:21.238117392 +0000 UTC m=+2160.856776557" watchObservedRunningTime="2026-10-31 13:39:21.241763821 +0000 UTC m=+2160.860423010"
Oct 31 13:39:25.213278 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:25.213236    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:39:27.230550 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:27.230510    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:39:27.824780 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:27.824743    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:27.839610 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:27.839579    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:28.310861 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:28.310826    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:39:34.701161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.701119    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"]
Oct 31 13:39:34.709743 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.708686    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.723502 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.720932    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cae9ec\"/\"default-dockercfg-mt6bc\""
Oct 31 13:39:34.723502 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.722052    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cae9ec\"/\"kube-root-ca.crt\""
Oct 31 13:39:34.723502 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.722111    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cae9ec\"/\"test-model-cae9ec-kserve-self-signed-certs\""
Oct 31 13:39:34.723502 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.722811    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cae9ec\"/\"openshift-service-ca.crt\""
Oct 31 13:39:34.728161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.728133    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"]
Oct 31 13:39:34.771915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.771858    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-model-cache\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.772118 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.771940    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/740f8e32-74e2-4c64-8d14-e0124a90bd5c-tls-certs\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.772118 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.772002    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-tmp-dir\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.772243 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.772172    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-dshm\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.772315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.772262    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqjjb\" (UniqueName: \"kubernetes.io/projected/740f8e32-74e2-4c64-8d14-e0124a90bd5c-kube-api-access-gqjjb\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.772381 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.772311    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-home\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.882135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.876614    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-tmp-dir\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.882135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.876813    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-dshm\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.882135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.876901    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqjjb\" (UniqueName: \"kubernetes.io/projected/740f8e32-74e2-4c64-8d14-e0124a90bd5c-kube-api-access-gqjjb\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.882135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.876933    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-home\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.882135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.877171    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-model-cache\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.882135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.877226    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/740f8e32-74e2-4c64-8d14-e0124a90bd5c-tls-certs\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.882135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.877958    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-tmp-dir\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.882135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.878216    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-home\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.882135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.878752    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-model-cache\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.882135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.879814    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-dshm\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.882135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.881911    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/740f8e32-74e2-4c64-8d14-e0124a90bd5c-tls-certs\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:34.899197 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:34.899085    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqjjb\" (UniqueName: \"kubernetes.io/projected/740f8e32-74e2-4c64-8d14-e0124a90bd5c-kube-api-access-gqjjb\") pod \"test-model-cae9ec-kserve-74f7f9945f-7795j\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") " pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:35.069368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:35.068644    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:35.514943 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:35.514397    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"]
Oct 31 13:39:35.524350 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:39:35.523618    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod740f8e32_74e2_4c64_8d14_e0124a90bd5c.slice/crio-6c3cd6139b6b2c3030fa7effc9b863232c875159becf006bf6fe9f189d6ed896 WatchSource:0}: Error finding container 6c3cd6139b6b2c3030fa7effc9b863232c875159becf006bf6fe9f189d6ed896: Status 404 returned error can't find the container with id 6c3cd6139b6b2c3030fa7effc9b863232c875159becf006bf6fe9f189d6ed896
Oct 31 13:39:36.459320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:36.458720    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j" event={"ID":"740f8e32-74e2-4c64-8d14-e0124a90bd5c","Type":"ContainerStarted","Data":"0683390beeea1aa3d67761048320fb291af582c54825a7100dd2294c2739a97d"}
Oct 31 13:39:36.459320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:36.458775    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j" event={"ID":"740f8e32-74e2-4c64-8d14-e0124a90bd5c","Type":"ContainerStarted","Data":"6c3cd6139b6b2c3030fa7effc9b863232c875159becf006bf6fe9f189d6ed896"}
Oct 31 13:39:36.485878 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:36.485809    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j" podStartSLOduration=2.485788558 podStartE2EDuration="2.485788558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:36.484201037 +0000 UTC m=+2176.102860200" watchObservedRunningTime="2026-10-31 13:39:36.485788558 +0000 UTC m=+2176.104447722"
Oct 31 13:39:37.069634 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.069586    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:37.228568 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.228093    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"]
Oct 31 13:39:37.235271 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.235121    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.247082 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.247044    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fb50fa\"/\"test-model-fb50fa-kserve-self-signed-certs\""
Oct 31 13:39:37.247536 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.247507    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fb50fa\"/\"openshift-service-ca.crt\""
Oct 31 13:39:37.247696 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.247677    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fb50fa\"/\"kube-root-ca.crt\""
Oct 31 13:39:37.248351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.248322    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fb50fa\"/\"default-dockercfg-r4ws4\""
Oct 31 13:39:37.252026 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.252002    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"]
Oct 31 13:39:37.429720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.429669    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrfq8\" (UniqueName: \"kubernetes.io/projected/13349545-8422-4466-9152-bd143810b334-kube-api-access-xrfq8\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.430062 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.430036    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-tmp-dir\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.430283 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.430264    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13349545-8422-4466-9152-bd143810b334-tls-certs\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.430588 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.430568    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-home\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.430853 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.430835    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-model-cache\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.430968 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.430954    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-dshm\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.567319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.561537    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-dshm\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.567319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.561659    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrfq8\" (UniqueName: \"kubernetes.io/projected/13349545-8422-4466-9152-bd143810b334-kube-api-access-xrfq8\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.567319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.562461    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-tmp-dir\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.567319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.562618    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13349545-8422-4466-9152-bd143810b334-tls-certs\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.567319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.562772    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-home\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.567319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.562955    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-model-cache\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.567319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.563369    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-model-cache\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.567319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.563802    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-tmp-dir\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.567319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.563871    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-home\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.585957 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.585414    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13349545-8422-4466-9152-bd143810b334-tls-certs\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.611331 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.586132    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-dshm\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.611331 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.594066    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrfq8\" (UniqueName: \"kubernetes.io/projected/13349545-8422-4466-9152-bd143810b334-kube-api-access-xrfq8\") pod \"test-model-fb50fa-kserve-548bdc8b6c-nj6kq\" (UID: \"13349545-8422-4466-9152-bd143810b334\") " pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:37.876438 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:37.873999    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:38.210217 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:39:38.210169    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13349545_8422_4466_9152_bd143810b334.slice/crio-763b32e545c2d204628b4c5ed3389402ebe5faaabcdd1c52d6e61a82de5d6f7d WatchSource:0}: Error finding container 763b32e545c2d204628b4c5ed3389402ebe5faaabcdd1c52d6e61a82de5d6f7d: Status 404 returned error can't find the container with id 763b32e545c2d204628b4c5ed3389402ebe5faaabcdd1c52d6e61a82de5d6f7d
Oct 31 13:39:38.211961 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:38.211937    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"]
Oct 31 13:39:38.534572 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:38.534479    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq" event={"ID":"13349545-8422-4466-9152-bd143810b334","Type":"ContainerStarted","Data":"f790a4b7f6c455619d4d9d3e7e5ac3770a55a4065698033f25c774766bd5b2e3"}
Oct 31 13:39:38.534572 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:38.534532    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq" event={"ID":"13349545-8422-4466-9152-bd143810b334","Type":"ContainerStarted","Data":"763b32e545c2d204628b4c5ed3389402ebe5faaabcdd1c52d6e61a82de5d6f7d"}
Oct 31 13:39:38.563989 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:38.563932    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq" podStartSLOduration=1.563917582 podStartE2EDuration="1.563917582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:38.562388783 +0000 UTC m=+2178.181047950" watchObservedRunningTime="2026-10-31 13:39:38.563917582 +0000 UTC m=+2178.182576746"
Oct 31 13:39:38.875096 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:38.875059    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:40.095110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.094500    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"]
Oct 31 13:39:40.109915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.109883    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.116193 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.116154    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"]
Oct 31 13:39:40.121685 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.120960    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-61f295\"/\"kube-root-ca.crt\""
Oct 31 13:39:40.121685 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.120996    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-61f295\"/\"openshift-service-ca.crt\""
Oct 31 13:39:40.124075 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.122282    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-61f295\"/\"default-dockercfg-fk9l9\""
Oct 31 13:39:40.124075 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.122464    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-61f295\"/\"test-model-61f295-kserve-self-signed-certs\""
Oct 31 13:39:40.239557 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.239518    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb9vs\" (UniqueName: \"kubernetes.io/projected/f3e60e1e-371d-4a37-b988-e76510d17e90-kube-api-access-tb9vs\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.239762 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.239725    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-model-cache\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.239837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.239762    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3e60e1e-371d-4a37-b988-e76510d17e90-tls-certs\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.239837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.239794    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-home\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.239837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.239822    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-tmp-dir\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.239989 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.239968    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-dshm\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.344922 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.344880    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb9vs\" (UniqueName: \"kubernetes.io/projected/f3e60e1e-371d-4a37-b988-e76510d17e90-kube-api-access-tb9vs\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.349400 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.345796    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-model-cache\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.350160 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.350017    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-model-cache\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.358098 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.357474    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3e60e1e-371d-4a37-b988-e76510d17e90-tls-certs\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.358098 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.357580    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-home\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.358098 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.357621    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-tmp-dir\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.358098 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.357858    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-dshm\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.358447 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.358338    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-home\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.370386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.362900    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-tmp-dir\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.396350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.384329    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-dshm\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.530318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.522077    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb9vs\" (UniqueName: \"kubernetes.io/projected/f3e60e1e-371d-4a37-b988-e76510d17e90-kube-api-access-tb9vs\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.530318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.526904    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3e60e1e-371d-4a37-b988-e76510d17e90-tls-certs\") pod \"test-model-61f295-kserve-798c45b56f-6sgqz\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") " pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:40.731240 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:40.731209    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:41.063053 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:41.063026    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"]
Oct 31 13:39:41.673308 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:41.673250    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz" event={"ID":"f3e60e1e-371d-4a37-b988-e76510d17e90","Type":"ContainerStarted","Data":"aff6c5fd25fb611d6782659427544f9f25c751a6c5d4e3ef8f4df514b3b5806b"}
Oct 31 13:39:41.673973 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:41.673945    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz" event={"ID":"f3e60e1e-371d-4a37-b988-e76510d17e90","Type":"ContainerStarted","Data":"86fe6f00b456b570a042017db6c69c878dbc5d584f92728a05de86331fe7f0ab"}
Oct 31 13:39:41.728387 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:41.709571    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz" podStartSLOduration=1.70955089 podStartE2EDuration="1.70955089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:41.707355325 +0000 UTC m=+2181.326014490" watchObservedRunningTime="2026-10-31 13:39:41.70955089 +0000 UTC m=+2181.328210058"
Oct 31 13:39:41.752319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:41.735448    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:45.105353 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:45.105316    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:45.449345 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:45.446858    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:45.807825 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:45.807717    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:39:47.877270 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:47.874994    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:47.911622 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:47.911591    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:48.886814 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:48.886777    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:39:50.736470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:50.735941    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:50.755422 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:39:50.755283    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:39:50.755732 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:50.755715    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:39:50.985383 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:39:50.985351    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:40:00.583698 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:00.583610    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"]
Oct 31 13:40:00.584204 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:00.583836    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy" containerID="cri-o://4d0d37f91315a91f0c49dc8c41bf59688e519435d7adab84f52db038476665ba" gracePeriod=30
Oct 31 13:40:01.651895 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:01.651852    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:40:01.652451 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:01.651933    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:40:06.304437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:06.304394    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:06.305800 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:06.305768    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:40:06.306246 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:06.306223    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:40:06.307331 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:06.307306    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm" podUID="cdc22df5-69c4-4def-9e7d-f022057cd410" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/ready\": context deadline exceeded"
Oct 31 13:40:06.366756 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:06.366320    4660 generic.go:356] "Generic (PLEG): container finished" podID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerID="4d0d37f91315a91f0c49dc8c41bf59688e519435d7adab84f52db038476665ba" exitCode=0
Oct 31 13:40:06.366756 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:06.366689    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" event={"ID":"4419c5c1-3c78-4b4f-afdc-17714fe39c54","Type":"ContainerDied","Data":"4d0d37f91315a91f0c49dc8c41bf59688e519435d7adab84f52db038476665ba"}
Oct 31 13:40:06.404411 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:06.404358    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:06.449464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:06.449411    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:06.472530 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:06.472483    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49" podUID="982ab639-4caa-445f-b029-4390d81cfad5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:06.614609 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:06.614499    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:06.686178 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:06.685148    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2" podUID="e003dfad-238c-47f3-aa19-696ca2c1264a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:06.806454 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:06.806404    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j" podUID="740f8e32-74e2-4c64-8d14-e0124a90bd5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:06.907819 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:06.898653    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq" podUID="13349545-8422-4466-9152-bd143810b334" containerName="main" probeResult="failure" output="Get \"https://10.133.0.141:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:07.493392 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.492812    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49" podUID="982ab639-4caa-445f-b029-4390d81cfad5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.131:8000/ready\": context deadline exceeded"
Oct 31 13:40:07.505327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.500095    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:07.555554 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.539893    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:40:07.587358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.557340    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:40:07.587358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.558001    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:40:07.587358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.558187    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:40:07.587358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.558251    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:40:07.589929 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.589873    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:40:07.781662 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.590476    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:40:07.887763 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.887724    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:40:07.887964 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.887872    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:40:07.896327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.598715    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm" podUID="cdc22df5-69c4-4def-9e7d-f022057cd410" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/ready\": context deadline exceeded"
Oct 31 13:40:07.896327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.702788    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2" podUID="e003dfad-238c-47f3-aa19-696ca2c1264a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:07.896327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.703001    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:40:07.936989 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.907446    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq" podUID="13349545-8422-4466-9152-bd143810b334" containerName="main" probeResult="failure" output="Get \"https://10.133.0.141:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:08.003362 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.991839    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:40:08.075118 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.075035    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:40:08.078644 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.078620    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:40:08.079428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.962841    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 31 13:40:08.079428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.752369    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 31 13:40:08.079428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.798786    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:40:08.079428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.079359    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk" podUID="77c1be44-f679-4953-9a0d-74f8c160e29c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:08.080047 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.590779    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:40:08.080047 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.807850    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j" podUID="740f8e32-74e2-4c64-8d14-e0124a90bd5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": context deadline exceeded"
Oct 31 13:40:08.080047 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.901973    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:40:08.080047 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.079874    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt" podUID="ede7a814-d7a4-44ef-b0b4-bf7377741d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:08.080047 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.995359    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:40:08.080047 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.079940    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9" podUID="fb081f1b-a184-4656-a834-240605f9693b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:08.080047 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:07.987652    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz" podUID="f3e60e1e-371d-4a37-b988-e76510d17e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:08.091569 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.091509    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:40:08.091724 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.091631    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:40:08.091724 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.091706    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:40:08.261699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.239651    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:40:08.261699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.239715    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:40:08.261699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.239748    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:40:08.261699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.239776    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:40:08.261699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.239801    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 13:40:08.261699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.239823    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:40:08.261699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.239852    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:40:08.261699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.260042    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 13:40:08.306036 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.278221    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw" event={"ID":"4419c5c1-3c78-4b4f-afdc-17714fe39c54","Type":"ContainerDied","Data":"c9415d4fd0548e963c75c1c3e406ed6b29bd8ee1bf2d83f1489518637d101022"}
Oct 31 13:40:08.306036 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.278285    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9415d4fd0548e963c75c1c3e406ed6b29bd8ee1bf2d83f1489518637d101022"
Oct 31 13:40:08.306036 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.282840    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:40:08.319045 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.318509    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 13:40:08.379852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.379813    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:40:08.390038 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.389985    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istiod-ca-cert\") pod \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") "
Oct 31 13:40:08.390350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.390320    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-podinfo\") pod \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") "
Oct 31 13:40:08.390525 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.390509    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4419c5c1-3c78-4b4f-afdc-17714fe39c54-kube-api-access-wdwhr\" (UniqueName: \"kubernetes.io/projected/4419c5c1-3c78-4b4f-afdc-17714fe39c54-kube-api-access-wdwhr\") pod \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") "
Oct 31 13:40:08.390724 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.390707    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-envoy\") pod \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") "
Oct 31 13:40:08.391153 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.391014    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-workload-socket\") pod \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") "
Oct 31 13:40:08.391435 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.391405    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-credential-socket\") pod \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") "
Oct 31 13:40:08.391594 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.391568    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-token\" (UniqueName: \"kubernetes.io/projected/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-token\") pod \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") "
Oct 31 13:40:08.391738 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.391723    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-data\") pod \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") "
Oct 31 13:40:08.391864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.391849    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-workload-certs\") pod \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\" (UID: \"4419c5c1-3c78-4b4f-afdc-17714fe39c54\") "
Oct 31 13:40:08.394247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.394212    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istiod-ca-cert" pod "4419c5c1-3c78-4b4f-afdc-17714fe39c54" (UID: "4419c5c1-3c78-4b4f-afdc-17714fe39c54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:40:08.430821 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.429497    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-workload-socket" pod "4419c5c1-3c78-4b4f-afdc-17714fe39c54" (UID: "4419c5c1-3c78-4b4f-afdc-17714fe39c54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:40:08.437437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.431689    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-workload-certs" pod "4419c5c1-3c78-4b4f-afdc-17714fe39c54" (UID: "4419c5c1-3c78-4b4f-afdc-17714fe39c54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:40:08.453526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.453433    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-credential-socket" pod "4419c5c1-3c78-4b4f-afdc-17714fe39c54" (UID: "4419c5c1-3c78-4b4f-afdc-17714fe39c54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:40:08.453526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.453499    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-data" pod "4419c5c1-3c78-4b4f-afdc-17714fe39c54" (UID: "4419c5c1-3c78-4b4f-afdc-17714fe39c54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:40:08.456367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.456285    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-envoy" pod "4419c5c1-3c78-4b4f-afdc-17714fe39c54" (UID: "4419c5c1-3c78-4b4f-afdc-17714fe39c54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:40:08.459049 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.459021    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-token" pod "4419c5c1-3c78-4b4f-afdc-17714fe39c54" (UID: "4419c5c1-3c78-4b4f-afdc-17714fe39c54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:40:08.462273 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.462249    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:40:08.476899 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.472422    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-podinfo" pod "4419c5c1-3c78-4b4f-afdc-17714fe39c54" (UID: "4419c5c1-3c78-4b4f-afdc-17714fe39c54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:40:08.478260 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.478232    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4419c5c1-3c78-4b4f-afdc-17714fe39c54-kube-api-access-wdwhr" pod "4419c5c1-3c78-4b4f-afdc-17714fe39c54" (UID: "4419c5c1-3c78-4b4f-afdc-17714fe39c54"). InnerVolumeSpecName "kube-api-access-wdwhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:40:08.495315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.495270    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:08.531986 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.531956    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:08.532107 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.532090    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdwhr\" (UniqueName: \"kubernetes.io/projected/4419c5c1-3c78-4b4f-afdc-17714fe39c54-kube-api-access-wdwhr\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:08.532201 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.532190    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:08.532282 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.532271    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:08.532411 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.532396    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:08.553317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.532493    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:08.553617 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.553585    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:08.553746 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.553728    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4419c5c1-3c78-4b4f-afdc-17714fe39c54-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:08.936343 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.922835    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"]
Oct 31 13:40:08.936343 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.924132    4660 cpu_manager.go:414] "containerMap: removing container" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy"
Oct 31 13:40:08.936343 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.924155    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy"
Oct 31 13:40:08.961351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.960221    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" containerName="istio-proxy"
Oct 31 13:40:08.971509 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.971476    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"]
Oct 31 13:40:08.971858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.971836    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:40:08.978760 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:08.978523    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-554759-serving-cert\""
Oct 31 13:40:09.079706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:09.075124    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3201b01f-5dd7-4338-aa51-6899d3ae4e64-maas-api-tls\") pod \"maas-api-e2e-inf-a-554759-d774c75df-8xqwm\" (UID: \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:40:09.080810 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:09.080779    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm4l7\" (UniqueName: \"kubernetes.io/projected/3201b01f-5dd7-4338-aa51-6899d3ae4e64-kube-api-access-jm4l7\") pod \"maas-api-e2e-inf-a-554759-d774c75df-8xqwm\" (UID: \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:40:09.081245 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:09.081227    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3201b01f-5dd7-4338-aa51-6899d3ae4e64-openshift-service-ca\") pod \"maas-api-e2e-inf-a-554759-d774c75df-8xqwm\" (UID: \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:40:09.098167 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:09.094278    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3201b01f-5dd7-4338-aa51-6899d3ae4e64-metrics-certs\") pod \"maas-api-e2e-inf-a-554759-d774c75df-8xqwm\" (UID: \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:40:09.229049 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:09.210903    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3201b01f-5dd7-4338-aa51-6899d3ae4e64-maas-api-tls\") pod \"maas-api-e2e-inf-a-554759-d774c75df-8xqwm\" (UID: \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:40:09.229049 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:09.210995    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm4l7\" (UniqueName: \"kubernetes.io/projected/3201b01f-5dd7-4338-aa51-6899d3ae4e64-kube-api-access-jm4l7\") pod \"maas-api-e2e-inf-a-554759-d774c75df-8xqwm\" (UID: \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:40:09.229049 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:09.211221    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3201b01f-5dd7-4338-aa51-6899d3ae4e64-openshift-service-ca\") pod \"maas-api-e2e-inf-a-554759-d774c75df-8xqwm\" (UID: \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:40:09.229049 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:09.211266    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3201b01f-5dd7-4338-aa51-6899d3ae4e64-metrics-certs\") pod \"maas-api-e2e-inf-a-554759-d774c75df-8xqwm\" (UID: \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:40:09.229049 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:09.212692    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3201b01f-5dd7-4338-aa51-6899d3ae4e64-openshift-service-ca\") pod \"maas-api-e2e-inf-a-554759-d774c75df-8xqwm\" (UID: \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:40:09.285062 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:09.285023    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm4l7\" (UniqueName: \"kubernetes.io/projected/3201b01f-5dd7-4338-aa51-6899d3ae4e64-kube-api-access-jm4l7\") pod \"maas-api-e2e-inf-a-554759-d774c75df-8xqwm\" (UID: \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:40:09.294531 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:09.294421    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"
Oct 31 13:40:09.310754 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:09.305422    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3201b01f-5dd7-4338-aa51-6899d3ae4e64-maas-api-tls\") pod \"maas-api-e2e-inf-a-554759-d774c75df-8xqwm\" (UID: \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:40:09.604152 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:09.584558    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3201b01f-5dd7-4338-aa51-6899d3ae4e64-metrics-certs\") pod \"maas-api-e2e-inf-a-554759-d774c75df-8xqwm\" (UID: \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:40:09.700894 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:09.700858    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:40:09.970408 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:09.970368    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"]
Oct 31 13:40:09.988433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:09.988366    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-kmwbw"]
Oct 31 13:40:10.409662 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:10.408440    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"]
Oct 31 13:40:10.635318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:10.633354    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4419c5c1-3c78-4b4f-afdc-17714fe39c54" path="/var/lib/kubelet/pods/4419c5c1-3c78-4b4f-afdc-17714fe39c54/volumes"
Oct 31 13:40:11.467502 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:11.467448    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm" event={"ID":"3201b01f-5dd7-4338-aa51-6899d3ae4e64","Type":"ContainerStarted","Data":"476ac7926d49521f758429523536f9357c8793356aa989c3cff5f8ad4cc68849"}
Oct 31 13:40:11.467713 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:11.467522    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm" event={"ID":"3201b01f-5dd7-4338-aa51-6899d3ae4e64","Type":"ContainerStarted","Data":"adeaa95cfde7e1c8daac99bebeb0f6919cbb0878d4e595313888d03a07a6dfd3"}
Oct 31 13:40:11.467713 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:11.467555    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:40:11.499280 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:11.499225    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm" podStartSLOduration=46.898478408 podStartE2EDuration="47.499209915s" totalImagesPullingTime="600.731507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:40:11.49185657 +0000 UTC m=+2211.110515737" watchObservedRunningTime="2026-10-31 13:40:11.499209915 +0000 UTC m=+2211.117869105"
Oct 31 13:40:13.892334 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:13.892270    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"]
Oct 31 13:40:13.907001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:13.905756    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:13.922801 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:13.922772    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0d835b\"/\"kube-root-ca.crt\""
Oct 31 13:40:13.923416 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:13.923200    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0d835b\"/\"test-model-0d835b-kserve-self-signed-certs\""
Oct 31 13:40:13.923680 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:13.923275    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0d835b\"/\"openshift-service-ca.crt\""
Oct 31 13:40:13.923784 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:13.923361    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0d835b\"/\"default-dockercfg-fbjb2\""
Oct 31 13:40:13.966566 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:13.966536    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"]
Oct 31 13:40:13.993057 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:13.993014    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-home\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:13.993257 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:13.993066    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-dshm\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:13.993257 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:13.993114    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-model-cache\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:13.993257 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:13.993138    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708ebdbd-2c8c-4d47-8d03-517e149613d1-tls-certs\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:13.994596 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:13.993517    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-tmp-dir\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:13.994596 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:13.993555    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdc99\" (UniqueName: \"kubernetes.io/projected/708ebdbd-2c8c-4d47-8d03-517e149613d1-kube-api-access-zdc99\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:14.097814 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:14.097772    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-tmp-dir\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:14.098091 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:14.098070    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdc99\" (UniqueName: \"kubernetes.io/projected/708ebdbd-2c8c-4d47-8d03-517e149613d1-kube-api-access-zdc99\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:14.098254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:14.098238    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-home\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:14.098401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:14.098383    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-dshm\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:14.102315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:14.098598    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-model-cache\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:14.102524 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:14.102500    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708ebdbd-2c8c-4d47-8d03-517e149613d1-tls-certs\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:14.127317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:14.121089    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708ebdbd-2c8c-4d47-8d03-517e149613d1-tls-certs\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:14.127317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:14.122316    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-tmp-dir\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:14.127317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:14.122921    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-home\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:14.127317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:14.124257    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-model-cache\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:14.131314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:14.128837    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-dshm\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:14.189368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:14.165417    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdc99\" (UniqueName: \"kubernetes.io/projected/708ebdbd-2c8c-4d47-8d03-517e149613d1-kube-api-access-zdc99\") pod \"test-model-0d835b-kserve-6c7564f9-mr5c4\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") " pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:14.246319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:14.243965    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:14.782068 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:40:14.782032    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod708ebdbd_2c8c_4d47_8d03_517e149613d1.slice/crio-ab163fef4618e3d4f4371181b657636653e6d4eec5fceac40d12e28f5bc9d921 WatchSource:0}: Error finding container ab163fef4618e3d4f4371181b657636653e6d4eec5fceac40d12e28f5bc9d921: Status 404 returned error can't find the container with id ab163fef4618e3d4f4371181b657636653e6d4eec5fceac40d12e28f5bc9d921
Oct 31 13:40:14.784403 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:14.783752    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"]
Oct 31 13:40:15.527077 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:15.526982    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4" event={"ID":"708ebdbd-2c8c-4d47-8d03-517e149613d1","Type":"ContainerStarted","Data":"5af5d7ce7ce3dc0238d7d601da92eb6817397903f5d626a24b791c1a8b5b8772"}
Oct 31 13:40:15.527077 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:15.527038    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4" event={"ID":"708ebdbd-2c8c-4d47-8d03-517e149613d1","Type":"ContainerStarted","Data":"ab163fef4618e3d4f4371181b657636653e6d4eec5fceac40d12e28f5bc9d921"}
Oct 31 13:40:15.552626 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:15.552554    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4" podStartSLOduration=2.552533011 podStartE2EDuration="2.552533011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:40:15.549820246 +0000 UTC m=+2215.168479411" watchObservedRunningTime="2026-10-31 13:40:15.552533011 +0000 UTC m=+2215.171192177"
Oct 31 13:40:16.245321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:16.245253    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:17.487761 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:17.487723    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:40:20.747317 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:40:20.747191    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:40:20.823507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:20.823467    4660 patch_prober.go:29] interesting pod/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:40:20.823676 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:20.823556    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr" podUID="d4e738ae-b6fd-4c4b-9413-0497e07cbc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:23.992890 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:23.992855    4660 scope.go:122] "RemoveContainer" containerID="4d0d37f91315a91f0c49dc8c41bf59688e519435d7adab84f52db038476665ba"
Oct 31 13:40:24.246962 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:24.245705    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:24.259626 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:24.259603    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:24.726318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:24.709610    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:40:28.786795 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:28.786760    4660 generic.go:356] "Generic (PLEG): container finished" podID="d4e738ae-b6fd-4c4b-9413-0497e07cbc03" containerID="1198ab63c389eb8eae6823862d0dd42d6609d985f42eb1d1e591615c7e77060e" exitCode=0
Oct 31 13:40:28.787256 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:28.786834    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr" event={"ID":"d4e738ae-b6fd-4c4b-9413-0497e07cbc03","Type":"ContainerDied","Data":"1198ab63c389eb8eae6823862d0dd42d6609d985f42eb1d1e591615c7e77060e"}
Oct 31 13:40:28.787492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:28.787473    4660 scope.go:122] "RemoveContainer" containerID="1198ab63c389eb8eae6823862d0dd42d6609d985f42eb1d1e591615c7e77060e"
Oct 31 13:40:28.789687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:28.789660    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:40:28.789768 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:28.789736    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:40:28.789808 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:28.789771    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:40:29.800965 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:29.800927    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr" event={"ID":"d4e738ae-b6fd-4c4b-9413-0497e07cbc03","Type":"ContainerStarted","Data":"fb5e316e3b7ccf232c86fae5cd889376172355f4737c9ec37fe99678a48ca19d"}
Oct 31 13:40:30.781840 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:30.781806    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:40:30.782814 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:30.782787    4660 patch_prober.go:29] interesting pod/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 31 13:40:30.782927 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:30.782859    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr" podUID="d4e738ae-b6fd-4c4b-9413-0497e07cbc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 31 13:40:31.782116 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:31.782078    4660 patch_prober.go:29] interesting pod/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 31 13:40:31.782548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:31.782147    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr" podUID="d4e738ae-b6fd-4c4b-9413-0497e07cbc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 31 13:40:32.781946 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:32.781903    4660 patch_prober.go:29] interesting pod/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 31 13:40:32.782122 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:32.781977    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr" podUID="d4e738ae-b6fd-4c4b-9413-0497e07cbc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 31 13:40:33.782341 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:33.782281    4660 patch_prober.go:29] interesting pod/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 31 13:40:33.782739 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:33.782374    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr" podUID="d4e738ae-b6fd-4c4b-9413-0497e07cbc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 31 13:40:34.781911 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:34.781877    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:40:34.782250 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:34.782223    4660 patch_prober.go:29] interesting pod/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 31 13:40:34.782481 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:34.782455    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr" podUID="d4e738ae-b6fd-4c4b-9413-0497e07cbc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 31 13:40:35.800367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:35.800321    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:40:35.802326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:35.802235    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:40:41.864214 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:41.864176    4660 patch_prober.go:29] interesting pod/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": read tcp 10.133.0.2:56832->10.133.0.133:15021: read: connection reset by peer" start-of-body=
Oct 31 13:40:41.865255 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:41.864260    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq" podUID="5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": read tcp 10.133.0.2:56832->10.133.0.133:15021: read: connection reset by peer"
Oct 31 13:40:42.030924 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:42.030842    4660 generic.go:356] "Generic (PLEG): container finished" podID="5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" containerID="59552b38c6f5b1cb11d559a1d59ba6b0dccdec5d8af4c337c73d3b98417a15c4" exitCode=0
Oct 31 13:40:42.031087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:42.030926    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq" event={"ID":"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e","Type":"ContainerDied","Data":"59552b38c6f5b1cb11d559a1d59ba6b0dccdec5d8af4c337c73d3b98417a15c4"}
Oct 31 13:40:42.031624 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:42.031597    4660 scope.go:122] "RemoveContainer" containerID="59552b38c6f5b1cb11d559a1d59ba6b0dccdec5d8af4c337c73d3b98417a15c4"
Oct 31 13:40:42.033956 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:42.033927    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:40:42.034021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:42.033999    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:40:42.034062 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:42.034037    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:40:43.047021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:43.046984    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq" event={"ID":"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e","Type":"ContainerStarted","Data":"5a54c535d6e375b40f60bc52e91b2a11adc24bea79905f516425402ae2e16688"}
Oct 31 13:40:43.864316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:43.859968    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:40:43.864316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:43.860106    4660 patch_prober.go:29] interesting pod/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 13:40:43.864316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:43.860159    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq" podUID="5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 13:40:44.850885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:44.850684    4660 patch_prober.go:29] interesting pod/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 13:40:44.850885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:44.850746    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq" podUID="5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 13:40:45.378540 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:45.378500    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"]
Oct 31 13:40:45.378759 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:45.378735    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" containerID="cri-o://3d418bde893a902938ba0daf48d9ec62bd6217fb88131bd8d9ee3bbcd16ac1c9" gracePeriod=30
Oct 31 13:40:45.449221 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:45.449153    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"]
Oct 31 13:40:45.449761 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:45.449724    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs" podUID="13e81951-495d-4169-bdbb-f6484e5d4756" containerName="istio-proxy" containerID="cri-o://d2b3f22c9fa201de3384b5beabbff18dce88667ca3462d7fecc720744cc758ca" gracePeriod=30
Oct 31 13:40:45.855667 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:45.855564    4660 patch_prober.go:29] interesting pod/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 13:40:45.863220 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:45.856276    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq" podUID="5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 13:40:46.166903 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:46.166866    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:40:46.167094 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:46.166927    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:40:46.850504 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:46.850464    4660 patch_prober.go:29] interesting pod/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 13:40:46.850709 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:46.850536    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq" podUID="5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 13:40:47.850719 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:47.850678    4660 patch_prober.go:29] interesting pod/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 13:40:47.851611 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:47.850746    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq" podUID="5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 13:40:48.856980 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:48.856941    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:40:48.857553 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:48.857533    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:40:48.858401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:48.858379    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:40:50.759863 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:40:50.759820    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:40:50.798496 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.798467    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:40:50.857214 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.857176    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"]
Oct 31 13:40:50.858269 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.858244    4660 cpu_manager.go:414] "containerMap: removing container" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy"
Oct 31 13:40:50.858269 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.858271    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy"
Oct 31 13:40:50.858614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.858577    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerName="istio-proxy"
Oct 31 13:40:50.863004 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.862983    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:50.874792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.874760    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"]
Oct 31 13:40:50.901644 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.901616    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:40:50.914930 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.914898    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-workload-certs\") pod \"6acc3b99-906c-429c-99f8-3aee4333eb66\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") "
Oct 31 13:40:50.915092 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.914952    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-envoy\") pod \"6acc3b99-906c-429c-99f8-3aee4333eb66\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") "
Oct 31 13:40:50.915092 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.914999    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-token\" (UniqueName: \"kubernetes.io/projected/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-token\") pod \"6acc3b99-906c-429c-99f8-3aee4333eb66\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") "
Oct 31 13:40:50.915219 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.915100    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-workload-socket\") pod \"6acc3b99-906c-429c-99f8-3aee4333eb66\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") "
Oct 31 13:40:50.915219 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.915153    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-data\") pod \"6acc3b99-906c-429c-99f8-3aee4333eb66\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") "
Oct 31 13:40:50.915219 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.915211    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-podinfo\") pod \"6acc3b99-906c-429c-99f8-3aee4333eb66\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") "
Oct 31 13:40:50.915501 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.915312    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6acc3b99-906c-429c-99f8-3aee4333eb66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6acc3b99-906c-429c-99f8-3aee4333eb66-istiod-ca-cert\") pod \"6acc3b99-906c-429c-99f8-3aee4333eb66\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") "
Oct 31 13:40:50.915501 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.915361    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-workload-certs" pod "6acc3b99-906c-429c-99f8-3aee4333eb66" (UID: "6acc3b99-906c-429c-99f8-3aee4333eb66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:40:50.915501 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.915432    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6acc3b99-906c-429c-99f8-3aee4333eb66-kube-api-access-f2zhh\" (UniqueName: \"kubernetes.io/projected/6acc3b99-906c-429c-99f8-3aee4333eb66-kube-api-access-f2zhh\") pod \"6acc3b99-906c-429c-99f8-3aee4333eb66\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") "
Oct 31 13:40:50.915663 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.915536    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-credential-socket\") pod \"6acc3b99-906c-429c-99f8-3aee4333eb66\" (UID: \"6acc3b99-906c-429c-99f8-3aee4333eb66\") "
Oct 31 13:40:50.915663 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.915574    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-workload-socket" pod "6acc3b99-906c-429c-99f8-3aee4333eb66" (UID: "6acc3b99-906c-429c-99f8-3aee4333eb66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:40:50.916413 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.916388    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:50.916525 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.916417    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:50.916657 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.916635    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-credential-socket" pod "6acc3b99-906c-429c-99f8-3aee4333eb66" (UID: "6acc3b99-906c-429c-99f8-3aee4333eb66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:40:50.917006 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.916989    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6acc3b99-906c-429c-99f8-3aee4333eb66-istiod-ca-cert" pod "6acc3b99-906c-429c-99f8-3aee4333eb66" (UID: "6acc3b99-906c-429c-99f8-3aee4333eb66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:40:50.917281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.917267    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-data" pod "6acc3b99-906c-429c-99f8-3aee4333eb66" (UID: "6acc3b99-906c-429c-99f8-3aee4333eb66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:40:50.918767 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.918749    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-token" pod "6acc3b99-906c-429c-99f8-3aee4333eb66" (UID: "6acc3b99-906c-429c-99f8-3aee4333eb66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:40:50.920000 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.919976    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-envoy" pod "6acc3b99-906c-429c-99f8-3aee4333eb66" (UID: "6acc3b99-906c-429c-99f8-3aee4333eb66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:40:50.921617 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.921594    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6acc3b99-906c-429c-99f8-3aee4333eb66-kube-api-access-f2zhh" pod "6acc3b99-906c-429c-99f8-3aee4333eb66" (UID: "6acc3b99-906c-429c-99f8-3aee4333eb66"). InnerVolumeSpecName "kube-api-access-f2zhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:40:50.922756 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.922736    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-podinfo" pod "6acc3b99-906c-429c-99f8-3aee4333eb66" (UID: "6acc3b99-906c-429c-99f8-3aee4333eb66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:40:50.961517 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.961458    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"]
Oct 31 13:40:50.967976 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.962970    4660 cpu_manager.go:414] "containerMap: removing container" podUID="13e81951-495d-4169-bdbb-f6484e5d4756" containerName="istio-proxy"
Oct 31 13:40:50.967976 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.962997    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13e81951-495d-4169-bdbb-f6484e5d4756" containerName="istio-proxy"
Oct 31 13:40:50.967976 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.963330    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="13e81951-495d-4169-bdbb-f6484e5d4756" containerName="istio-proxy"
Oct 31 13:40:50.972912 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.972886    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:50.987883 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.987779    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"]
Oct 31 13:40:50.996499 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:50.994788    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-35b584-openshift-default-dockercfg-ms4g9\""
Oct 31 13:40:51.024245 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.024211    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e81951-495d-4169-bdbb-f6484e5d4756-istio-token\" (UniqueName: \"kubernetes.io/projected/13e81951-495d-4169-bdbb-f6484e5d4756-istio-token\") pod \"13e81951-495d-4169-bdbb-f6484e5d4756\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") "
Oct 31 13:40:51.024652 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.024630    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-workload-certs\") pod \"13e81951-495d-4169-bdbb-f6484e5d4756\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") "
Oct 31 13:40:51.026640 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.026514    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-workload-certs" pod "13e81951-495d-4169-bdbb-f6484e5d4756" (UID: "13e81951-495d-4169-bdbb-f6484e5d4756"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:40:51.027085 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.027067    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e81951-495d-4169-bdbb-f6484e5d4756-kube-api-access-zd5f9\" (UniqueName: \"kubernetes.io/projected/13e81951-495d-4169-bdbb-f6484e5d4756-kube-api-access-zd5f9\") pod \"13e81951-495d-4169-bdbb-f6484e5d4756\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") "
Oct 31 13:40:51.027266 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.027250    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-istio-envoy\") pod \"13e81951-495d-4169-bdbb-f6484e5d4756\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") "
Oct 31 13:40:51.028144 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.028102    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-workload-socket" pod "13e81951-495d-4169-bdbb-f6484e5d4756" (UID: "13e81951-495d-4169-bdbb-f6484e5d4756"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:40:51.028859 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.028838    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-workload-socket\") pod \"13e81951-495d-4169-bdbb-f6484e5d4756\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") "
Oct 31 13:40:51.029070 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.029037    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e81951-495d-4169-bdbb-f6484e5d4756-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e81951-495d-4169-bdbb-f6484e5d4756-istiod-ca-cert\") pod \"13e81951-495d-4169-bdbb-f6484e5d4756\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") "
Oct 31 13:40:51.029154 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.029113    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-istio-data\") pod \"13e81951-495d-4169-bdbb-f6484e5d4756\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") "
Oct 31 13:40:51.029276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.029262    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-credential-socket\") pod \"13e81951-495d-4169-bdbb-f6484e5d4756\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") "
Oct 31 13:40:51.029722 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.029670    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-credential-socket" pod "13e81951-495d-4169-bdbb-f6484e5d4756" (UID: "13e81951-495d-4169-bdbb-f6484e5d4756"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:40:51.030015 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.029983    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-istio-data" pod "13e81951-495d-4169-bdbb-f6484e5d4756" (UID: "13e81951-495d-4169-bdbb-f6484e5d4756"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:40:51.030419 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.030390    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e81951-495d-4169-bdbb-f6484e5d4756-istiod-ca-cert" pod "13e81951-495d-4169-bdbb-f6484e5d4756" (UID: "13e81951-495d-4169-bdbb-f6484e5d4756"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:40:51.041729 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.041625    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13e81951-495d-4169-bdbb-f6484e5d4756-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e81951-495d-4169-bdbb-f6484e5d4756-istio-podinfo\") pod \"13e81951-495d-4169-bdbb-f6484e5d4756\" (UID: \"13e81951-495d-4169-bdbb-f6484e5d4756\") "
Oct 31 13:40:51.042703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.042656    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-podinfo\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.042915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.042893    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bfj8\" (UniqueName: \"kubernetes.io/projected/17767cc6-6a5e-4bb9-84d2-98162742cd43-kube-api-access-9bfj8\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.043088 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.043063    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.043393 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.043364    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.043536 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.043515    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecef293-ce25-4fc8-84cf-91140172b297-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.043612 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.043591    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecef293-ce25-4fc8-84cf-91140172b297-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.043672 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.043646    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-credential-socket\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.043739 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.043723    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.043796 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.043757    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-data\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.043852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.043798    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.043852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.043829    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-token\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.043957 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.043940    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecef293-ce25-4fc8-84cf-91140172b297-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.044012 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.043998    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-envoy\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.044065 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044043    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-workload-certs\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.044121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044088    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.044176 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044155    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqtlr\" (UniqueName: \"kubernetes.io/projected/fecef293-ce25-4fc8-84cf-91140172b297-kube-api-access-hqtlr\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.044315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044223    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-workload-socket\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.044315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044262    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17767cc6-6a5e-4bb9-84d2-98162742cd43-istiod-ca-cert\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.044427 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044411    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:51.044496 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044436    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:51.044496 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044452    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:51.044496 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044467    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:51.044496 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044482    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6acc3b99-906c-429c-99f8-3aee4333eb66-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:51.044703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044498    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2zhh\" (UniqueName: \"kubernetes.io/projected/6acc3b99-906c-429c-99f8-3aee4333eb66-kube-api-access-f2zhh\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:51.044703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044515    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:51.044703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044531    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:51.044703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044545    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:51.044703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044560    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:51.044703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044576    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e81951-495d-4169-bdbb-f6484e5d4756-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:51.044703 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.044590    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6acc3b99-906c-429c-99f8-3aee4333eb66-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:51.096346 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.094837    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-istio-envoy" pod "13e81951-495d-4169-bdbb-f6484e5d4756" (UID: "13e81951-495d-4169-bdbb-f6484e5d4756"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:40:51.096775 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.096746    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13e81951-495d-4169-bdbb-f6484e5d4756-istio-podinfo" pod "13e81951-495d-4169-bdbb-f6484e5d4756" (UID: "13e81951-495d-4169-bdbb-f6484e5d4756"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:40:51.100988 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.100933    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e81951-495d-4169-bdbb-f6484e5d4756-istio-token" pod "13e81951-495d-4169-bdbb-f6484e5d4756" (UID: "13e81951-495d-4169-bdbb-f6484e5d4756"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:40:51.103692 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.103663    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e81951-495d-4169-bdbb-f6484e5d4756-kube-api-access-zd5f9" pod "13e81951-495d-4169-bdbb-f6484e5d4756" (UID: "13e81951-495d-4169-bdbb-f6484e5d4756"). InnerVolumeSpecName "kube-api-access-zd5f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:40:51.133619 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.133580    4660 generic.go:356] "Generic (PLEG): container finished" podID="6acc3b99-906c-429c-99f8-3aee4333eb66" containerID="3d418bde893a902938ba0daf48d9ec62bd6217fb88131bd8d9ee3bbcd16ac1c9" exitCode=0
Oct 31 13:40:51.133837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.133657    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"
Oct 31 13:40:51.133837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.133682    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" event={"ID":"6acc3b99-906c-429c-99f8-3aee4333eb66","Type":"ContainerDied","Data":"3d418bde893a902938ba0daf48d9ec62bd6217fb88131bd8d9ee3bbcd16ac1c9"}
Oct 31 13:40:51.133837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.133752    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn" event={"ID":"6acc3b99-906c-429c-99f8-3aee4333eb66","Type":"ContainerDied","Data":"073caf68985e2a7da2f9c3c6c7e1e78d0a4ad852a3605a8a753f0c13ea54ed7b"}
Oct 31 13:40:51.133837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.133779    4660 scope.go:122] "RemoveContainer" containerID="3d418bde893a902938ba0daf48d9ec62bd6217fb88131bd8d9ee3bbcd16ac1c9"
Oct 31 13:40:51.136511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.136478    4660 generic.go:356] "Generic (PLEG): container finished" podID="13e81951-495d-4169-bdbb-f6484e5d4756" containerID="d2b3f22c9fa201de3384b5beabbff18dce88667ca3462d7fecc720744cc758ca" exitCode=0
Oct 31 13:40:51.136650 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.136532    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs" event={"ID":"13e81951-495d-4169-bdbb-f6484e5d4756","Type":"ContainerDied","Data":"d2b3f22c9fa201de3384b5beabbff18dce88667ca3462d7fecc720744cc758ca"}
Oct 31 13:40:51.136650 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.136578    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs" event={"ID":"13e81951-495d-4169-bdbb-f6484e5d4756","Type":"ContainerDied","Data":"56c6187b609457ce77cf3fd6ec150a64d15cf2b320bd310a49e1159f1d212609"}
Oct 31 13:40:51.136759 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.136653    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"
Oct 31 13:40:51.147823 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.147789    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqtlr\" (UniqueName: \"kubernetes.io/projected/fecef293-ce25-4fc8-84cf-91140172b297-kube-api-access-hqtlr\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.148071 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.148022    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-workload-socket\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.148994 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.148945    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-workload-socket\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.149121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.148551    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17767cc6-6a5e-4bb9-84d2-98162742cd43-istiod-ca-cert\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.149349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.149332    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-podinfo\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.149467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.149452    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bfj8\" (UniqueName: \"kubernetes.io/projected/17767cc6-6a5e-4bb9-84d2-98162742cd43-kube-api-access-9bfj8\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.149585 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.149571    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.149874 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.149856    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.150076 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.150061    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecef293-ce25-4fc8-84cf-91140172b297-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.150246 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.150220    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecef293-ce25-4fc8-84cf-91140172b297-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.150503 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.150486    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-credential-socket\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.150692 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.150678    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.150848 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.150794    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-data\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.151010 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.150834    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.151010 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.150961    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-token\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.151249 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.151234    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecef293-ce25-4fc8-84cf-91140172b297-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.151411 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.151389    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-envoy\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.151565 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.151528    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-workload-certs\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.151745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.151678    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.152011 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.151997    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e81951-495d-4169-bdbb-f6484e5d4756-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:51.152128 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.152117    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e81951-495d-4169-bdbb-f6484e5d4756-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:51.152281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.152221    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd5f9\" (UniqueName: \"kubernetes.io/projected/13e81951-495d-4169-bdbb-f6484e5d4756-kube-api-access-zd5f9\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:51.152281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.152239    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e81951-495d-4169-bdbb-f6484e5d4756-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:40:51.152850 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.152833    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.154353 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.154332    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.156345 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.155428    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-data\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.156345 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.155722    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.156345 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.156217    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17767cc6-6a5e-4bb9-84d2-98162742cd43-istiod-ca-cert\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.160214 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.159822    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.162333 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.162165    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecef293-ce25-4fc8-84cf-91140172b297-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.162671 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.162633    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-podinfo\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.162770 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.162709    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqtlr\" (UniqueName: \"kubernetes.io/projected/fecef293-ce25-4fc8-84cf-91140172b297-kube-api-access-hqtlr\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.163071 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.162957    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-credential-socket\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.163313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.163261    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-envoy\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.163974 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.163953    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-workload-certs\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.165721 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.165484    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.165721 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.165672    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecef293-ce25-4fc8-84cf-91140172b297-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.170285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.169936    4660 scope.go:122] "RemoveContainer" containerID="3d418bde893a902938ba0daf48d9ec62bd6217fb88131bd8d9ee3bbcd16ac1c9"
Oct 31 13:40:51.170617 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:40:51.170562    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d418bde893a902938ba0daf48d9ec62bd6217fb88131bd8d9ee3bbcd16ac1c9\": container with ID starting with 3d418bde893a902938ba0daf48d9ec62bd6217fb88131bd8d9ee3bbcd16ac1c9 not found: ID does not exist" containerID="3d418bde893a902938ba0daf48d9ec62bd6217fb88131bd8d9ee3bbcd16ac1c9"
Oct 31 13:40:51.170617 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.170593    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d418bde893a902938ba0daf48d9ec62bd6217fb88131bd8d9ee3bbcd16ac1c9"} err="failed to get container status \"3d418bde893a902938ba0daf48d9ec62bd6217fb88131bd8d9ee3bbcd16ac1c9\": rpc error: code = NotFound desc = could not find container \"3d418bde893a902938ba0daf48d9ec62bd6217fb88131bd8d9ee3bbcd16ac1c9\": container with ID starting with 3d418bde893a902938ba0daf48d9ec62bd6217fb88131bd8d9ee3bbcd16ac1c9 not found: ID does not exist"
Oct 31 13:40:51.170617 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.170616    4660 scope.go:122] "RemoveContainer" containerID="d2b3f22c9fa201de3384b5beabbff18dce88667ca3462d7fecc720744cc758ca"
Oct 31 13:40:51.172148 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.172123    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-token\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.173076 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.173048    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecef293-ce25-4fc8-84cf-91140172b297-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.174153 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.174134    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bfj8\" (UniqueName: \"kubernetes.io/projected/17767cc6-6a5e-4bb9-84d2-98162742cd43-kube-api-access-9bfj8\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-2zj26\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.188355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.188219    4660 scope.go:122] "RemoveContainer" containerID="d2b3f22c9fa201de3384b5beabbff18dce88667ca3462d7fecc720744cc758ca"
Oct 31 13:40:51.188661 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:40:51.188636    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b3f22c9fa201de3384b5beabbff18dce88667ca3462d7fecc720744cc758ca\": container with ID starting with d2b3f22c9fa201de3384b5beabbff18dce88667ca3462d7fecc720744cc758ca not found: ID does not exist" containerID="d2b3f22c9fa201de3384b5beabbff18dce88667ca3462d7fecc720744cc758ca"
Oct 31 13:40:51.188750 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.188669    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b3f22c9fa201de3384b5beabbff18dce88667ca3462d7fecc720744cc758ca"} err="failed to get container status \"d2b3f22c9fa201de3384b5beabbff18dce88667ca3462d7fecc720744cc758ca\": rpc error: code = NotFound desc = could not find container \"d2b3f22c9fa201de3384b5beabbff18dce88667ca3462d7fecc720744cc758ca\": container with ID starting with d2b3f22c9fa201de3384b5beabbff18dce88667ca3462d7fecc720744cc758ca not found: ID does not exist"
Oct 31 13:40:51.194845 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.194809    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"]
Oct 31 13:40:51.200506 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.200481    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-7bghs"]
Oct 31 13:40:51.213465 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.213421    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"]
Oct 31 13:40:51.215107 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.215084    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:51.221945 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.221918    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-8btzn"]
Oct 31 13:40:51.309238 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.309195    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:51.520912 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:40:51.520883    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecef293_ce25_4fc8_84cf_91140172b297.slice/crio-20e3cd5c876065ca4fb1a9b56019a2506108242398de70b011564263a671f8c8 WatchSource:0}: Error finding container 20e3cd5c876065ca4fb1a9b56019a2506108242398de70b011564263a671f8c8: Status 404 returned error can't find the container with id 20e3cd5c876065ca4fb1a9b56019a2506108242398de70b011564263a671f8c8
Oct 31 13:40:51.525484 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.525439    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:40:51.525712 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.525679    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:40:51.525898 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.525877    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:40:51.534086 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.534061    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"]
Oct 31 13:40:51.632175 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.632133    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"]
Oct 31 13:40:51.635634 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.635577    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:40:51.635736 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.635679    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:40:51.635808 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:51.635739    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:40:52.160356 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:52.157377    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26" event={"ID":"17767cc6-6a5e-4bb9-84d2-98162742cd43","Type":"ContainerStarted","Data":"42fc1641631e9d3f537382a83757b92c04964049e1ebe23efd0cec61d78408b8"}
Oct 31 13:40:52.160356 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:52.157477    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26" event={"ID":"17767cc6-6a5e-4bb9-84d2-98162742cd43","Type":"ContainerStarted","Data":"a8b5ef69faaf065f03294c5acd285cc52f40ebb1a0ba4c9ab43d7e16e9934d70"}
Oct 31 13:40:52.163457 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:52.163410    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6" event={"ID":"fecef293-ce25-4fc8-84cf-91140172b297","Type":"ContainerStarted","Data":"66382f62e91bbbb7698cc561d8e1d3d88decadb0cf17fa2f01b1951b7fd4b331"}
Oct 31 13:40:52.163744 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:52.163684    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6" event={"ID":"fecef293-ce25-4fc8-84cf-91140172b297","Type":"ContainerStarted","Data":"20e3cd5c876065ca4fb1a9b56019a2506108242398de70b011564263a671f8c8"}
Oct 31 13:40:52.206373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:52.206318    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26" podStartSLOduration=27.20628823 podStartE2EDuration="27.20628823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:40:52.199726522 +0000 UTC m=+2251.818385687" watchObservedRunningTime="2026-10-31 13:40:52.20628823 +0000 UTC m=+2251.824947393"
Oct 31 13:40:52.215771 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:52.215727    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:52.217387 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:52.217358    4660 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 31 13:40:52.217525 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:52.217418    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6" podUID="fecef293-ce25-4fc8-84cf-91140172b297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 31 13:40:52.240449 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:52.240396    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6" podStartSLOduration=86.240381963 podStartE2EDuration="1m26.240381963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:40:52.23694417 +0000 UTC m=+2251.855603358" watchObservedRunningTime="2026-10-31 13:40:52.240381963 +0000 UTC m=+2251.859041126"
Oct 31 13:40:52.317729 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:52.317680    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:52.386936 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:52.386908    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:52.564304 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:52.564188    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13e81951-495d-4169-bdbb-f6484e5d4756" path="/var/lib/kubelet/pods/13e81951-495d-4169-bdbb-f6484e5d4756/volumes"
Oct 31 13:40:52.565215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:52.565189    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6acc3b99-906c-429c-99f8-3aee4333eb66" path="/var/lib/kubelet/pods/6acc3b99-906c-429c-99f8-3aee4333eb66/volumes"
Oct 31 13:40:53.178046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:53.178013    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:53.180439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:53.180416    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:40:53.223349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:53.221705    4660 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 31 13:40:53.223349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:53.221778    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6" podUID="fecef293-ce25-4fc8-84cf-91140172b297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 31 13:40:54.215794 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:54.215754    4660 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 31 13:40:54.216425 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:54.216390    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6" podUID="fecef293-ce25-4fc8-84cf-91140172b297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 31 13:40:55.237319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:55.236920    4660 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 31 13:40:55.237319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:55.236987    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6" podUID="fecef293-ce25-4fc8-84cf-91140172b297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 31 13:40:56.253456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:56.252341    4660 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 31 13:40:56.253456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:56.252402    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6" podUID="fecef293-ce25-4fc8-84cf-91140172b297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 31 13:40:57.220584 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:57.220546    4660 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 31 13:40:57.220887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:57.220857    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6" podUID="fecef293-ce25-4fc8-84cf-91140172b297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 31 13:40:58.215759 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:58.215716    4660 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 31 13:40:58.216179 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:58.215777    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6" podUID="fecef293-ce25-4fc8-84cf-91140172b297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 31 13:40:59.222311 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:59.222260    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:59.222988 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:59.222954    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:40:59.223669 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:40:59.223651    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:41:22.038114 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:21.977802    4660 patch_prober.go:29] interesting pod/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:41:22.192035 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:41:21.997424    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:41:22.192035 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:22.150728    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm" podUID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:41:23.895730 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:23.895686    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw" podUID="84174b11-1866-4f83-ad10-84a6288b53dd" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.59:50052\" within 1s: context deadline exceeded"
Oct 31 13:41:23.902016 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:41:23.901983    4660 logging.go:55] [core] [Channel #329 SubChannel #330]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.59:50052", ServerName: "10.133.0.59:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.59:50052: operation was canceled"
Oct 31 13:41:26.893441 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:26.893402    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm" event={"ID":"8a52dcc2-c2ab-430c-988e-d3b967f833f3","Type":"ContainerDied","Data":"e365c0fcce366ead8f5df728c37af26995d99ecaf27ad58f9b5e2ecd7c4644fc"}
Oct 31 13:41:26.894164 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:26.893787    4660 generic.go:356] "Generic (PLEG): container finished" podID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" containerID="e365c0fcce366ead8f5df728c37af26995d99ecaf27ad58f9b5e2ecd7c4644fc" exitCode=0
Oct 31 13:41:26.895585 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:26.895562    4660 scope.go:122] "RemoveContainer" containerID="e365c0fcce366ead8f5df728c37af26995d99ecaf27ad58f9b5e2ecd7c4644fc"
Oct 31 13:41:26.899318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:26.899274    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:41:26.900006 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:26.899989    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:41:26.900055 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:26.900032    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:41:27.938317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:27.932994    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm" event={"ID":"8a52dcc2-c2ab-430c-988e-d3b967f833f3","Type":"ContainerStarted","Data":"c74014eac6f8f8891971e981ce3daaa2cade9a001608317240864b7f9df65f1c"}
Oct 31 13:41:27.956531 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:27.956423    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:41:28.956990 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:28.956928    4660 patch_prober.go:29] interesting pod/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 31 13:41:28.957422 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:28.957000    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm" podUID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 31 13:41:29.956789 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:29.956741    4660 patch_prober.go:29] interesting pod/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 31 13:41:29.956982 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:29.956807    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm" podUID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 31 13:41:30.957772 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:30.957723    4660 patch_prober.go:29] interesting pod/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 31 13:41:30.958515 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:30.958479    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm" podUID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 31 13:41:31.957130 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:31.956685    4660 patch_prober.go:29] interesting pod/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 31 13:41:31.957130 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:31.956746    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm" podUID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 31 13:41:32.956741 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:32.956683    4660 patch_prober.go:29] interesting pod/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 31 13:41:32.957268 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:32.956755    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm" podUID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 31 13:41:33.962864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:33.962829    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:41:33.963848 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:33.963324    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:41:33.963943 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:41:33.963906    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:41:50.736232 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:41:50.736185    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:42:14.360801 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.360768    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"]
Oct 31 13:42:14.498911 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.498879    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.520509 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.520476    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"]
Oct 31 13:42:14.560846 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.560813    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-554759\"/\"default-dockercfg-fr695\""
Oct 31 13:42:14.605624 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.570831    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-554759\"/\"test-model-554759-kserve-self-signed-certs\""
Oct 31 13:42:14.605624 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.571925    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-554759\"/\"kube-root-ca.crt\""
Oct 31 13:42:14.605624 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.572020    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-554759\"/\"openshift-service-ca.crt\""
Oct 31 13:42:14.767433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.767389    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-dshm\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.767646 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.767518    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59caaca5-1f5a-42fc-841d-234c76fef93e-tls-certs\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.767646 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.767641    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-model-cache\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.767777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.767723    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-tmp-dir\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.768000 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.767983    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-home\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.768065 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.768024    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfft5\" (UniqueName: \"kubernetes.io/projected/59caaca5-1f5a-42fc-841d-234c76fef93e-kube-api-access-vfft5\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.870165 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.870105    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-home\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.870931 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.870391    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfft5\" (UniqueName: \"kubernetes.io/projected/59caaca5-1f5a-42fc-841d-234c76fef93e-kube-api-access-vfft5\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.870931 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.870565    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-dshm\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.870931 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.870642    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59caaca5-1f5a-42fc-841d-234c76fef93e-tls-certs\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.870931 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.870740    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-model-cache\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.871252 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.870803    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-tmp-dir\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.873491 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.873433    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-model-cache\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.873790 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.873769    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-home\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.874128 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.874106    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-tmp-dir\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.880017 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.879994    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-dshm\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.884341 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.884279    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59caaca5-1f5a-42fc-841d-234c76fef93e-tls-certs\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:14.887858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:14.887812    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfft5\" (UniqueName: \"kubernetes.io/projected/59caaca5-1f5a-42fc-841d-234c76fef93e-kube-api-access-vfft5\") pod \"test-model-554759-kserve-7c7c744997-p9584\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") " pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:15.007020 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:15.006605    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:15.745483 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:15.744853    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"]
Oct 31 13:42:15.748478 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:42:15.747357    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59caaca5_1f5a_42fc_841d_234c76fef93e.slice/crio-9eb33216af8981cc802c5c565019963db10b0270c08026496defb5d8e400526a WatchSource:0}: Error finding container 9eb33216af8981cc802c5c565019963db10b0270c08026496defb5d8e400526a: Status 404 returned error can't find the container with id 9eb33216af8981cc802c5c565019963db10b0270c08026496defb5d8e400526a
Oct 31 13:42:15.849353 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:15.849195    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584" event={"ID":"59caaca5-1f5a-42fc-841d-234c76fef93e","Type":"ContainerStarted","Data":"9eb33216af8981cc802c5c565019963db10b0270c08026496defb5d8e400526a"}
Oct 31 13:42:16.862591 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:16.861722    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584" event={"ID":"59caaca5-1f5a-42fc-841d-234c76fef93e","Type":"ContainerStarted","Data":"7f06008b50b7f2e9cdbbd016cc5b477e7aca71d4080e09ab3953545ad698b42f"}
Oct 31 13:42:16.886946 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:16.886892    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584" podStartSLOduration=3.886876054 podStartE2EDuration="3.886876054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:42:16.883377237 +0000 UTC m=+2336.502036401" watchObservedRunningTime="2026-10-31 13:42:16.886876054 +0000 UTC m=+2336.505535221"
Oct 31 13:42:17.009497 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:17.009463    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:20.747054 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:42:20.747013    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:42:24.219536 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:24.219418    4660 patch_prober.go:29] interesting pod/e2e-inf-b-35b584-openshift-default-556b575787-2zj26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:42:24.220246 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:24.220183    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26" podUID="17767cc6-6a5e-4bb9-84d2-98162742cd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:42:25.007180 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:25.007141    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:25.020862 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:25.020830    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:26.018220 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:26.018191    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:42:27.018084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:27.018046    4660 generic.go:356] "Generic (PLEG): container finished" podID="17767cc6-6a5e-4bb9-84d2-98162742cd43" containerID="42fc1641631e9d3f537382a83757b92c04964049e1ebe23efd0cec61d78408b8" exitCode=0
Oct 31 13:42:27.018282 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:27.018083    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26" event={"ID":"17767cc6-6a5e-4bb9-84d2-98162742cd43","Type":"ContainerDied","Data":"42fc1641631e9d3f537382a83757b92c04964049e1ebe23efd0cec61d78408b8"}
Oct 31 13:42:27.018718 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:27.018618    4660 scope.go:122] "RemoveContainer" containerID="42fc1641631e9d3f537382a83757b92c04964049e1ebe23efd0cec61d78408b8"
Oct 31 13:42:27.021323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:27.021274    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:42:27.021431 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:27.021383    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:42:27.021484 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:27.021428    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:42:28.051660 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:28.051624    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26" event={"ID":"17767cc6-6a5e-4bb9-84d2-98162742cd43","Type":"ContainerStarted","Data":"f2283b1a9fe3d8a84916184367cf889129bd985ebc494cc89c2ff9f3b6edeb20"}
Oct 31 13:42:28.316530 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:28.316432    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:42:28.317352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:28.317158    4660 patch_prober.go:29] interesting pod/e2e-inf-b-35b584-openshift-default-556b575787-2zj26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 31 13:42:28.317571 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:28.317546    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26" podUID="17767cc6-6a5e-4bb9-84d2-98162742cd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 31 13:42:29.327661 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:29.327628    4660 patch_prober.go:29] interesting pod/e2e-inf-b-35b584-openshift-default-556b575787-2zj26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 31 13:42:29.328255 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:29.328221    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26" podUID="17767cc6-6a5e-4bb9-84d2-98162742cd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 31 13:42:30.310691 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:30.310652    4660 patch_prober.go:29] interesting pod/e2e-inf-b-35b584-openshift-default-556b575787-2zj26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 31 13:42:30.310990 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:30.310941    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26" podUID="17767cc6-6a5e-4bb9-84d2-98162742cd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 31 13:42:31.310705 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:31.310664    4660 patch_prober.go:29] interesting pod/e2e-inf-b-35b584-openshift-default-556b575787-2zj26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 31 13:42:31.314447 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:31.311216    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26" podUID="17767cc6-6a5e-4bb9-84d2-98162742cd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 31 13:42:32.322318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.321685    4660 patch_prober.go:29] interesting pod/e2e-inf-b-35b584-openshift-default-556b575787-2zj26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 31 13:42:32.322318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.321780    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26" podUID="17767cc6-6a5e-4bb9-84d2-98162742cd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 31 13:42:32.595642 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.594491    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"]
Oct 31 13:42:32.657501 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.657467    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.666732 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.666380    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"]
Oct 31 13:42:32.670251 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.670227    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-35b584\"/\"openshift-service-ca.crt\""
Oct 31 13:42:32.675578 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.672672    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-35b584\"/\"kube-root-ca.crt\""
Oct 31 13:42:32.675767 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.672959    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-35b584\"/\"test-model-35b584-kserve-self-signed-certs\""
Oct 31 13:42:32.675767 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.673059    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-35b584\"/\"default-dockercfg-htl2l\""
Oct 31 13:42:32.755817 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.755774    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-model-cache\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.756155 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.756130    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4k8b\" (UniqueName: \"kubernetes.io/projected/a5fe2659-7755-4acc-adcd-481f1d5fa982-kube-api-access-w4k8b\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.756413 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.756395    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fe2659-7755-4acc-adcd-481f1d5fa982-tls-certs\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.756625 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.756592    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-tmp-dir\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.756754 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.756739    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-home\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.756949 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.756929    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-dshm\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.858592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.858510    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fe2659-7755-4acc-adcd-481f1d5fa982-tls-certs\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.858759 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.858601    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-tmp-dir\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.858759 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.858635    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-home\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.858759 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.858718    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-dshm\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.859767 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.859740    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-tmp-dir\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.859993 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.859967    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-model-cache\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.860098 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.860079    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4k8b\" (UniqueName: \"kubernetes.io/projected/a5fe2659-7755-4acc-adcd-481f1d5fa982-kube-api-access-w4k8b\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.860737 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.860706    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-model-cache\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.861137 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.861114    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-home\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.863739 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.863716    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fe2659-7755-4acc-adcd-481f1d5fa982-tls-certs\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.880516 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.880443    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4k8b\" (UniqueName: \"kubernetes.io/projected/a5fe2659-7755-4acc-adcd-481f1d5fa982-kube-api-access-w4k8b\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.887760 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.887719    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-dshm\") pod \"test-model-35b584-kserve-77fb974cb5-mfj6g\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") " pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:32.987313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:32.986663    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:33.310318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:33.309982    4660 patch_prober.go:29] interesting pod/e2e-inf-b-35b584-openshift-default-556b575787-2zj26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 31 13:42:33.310318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:33.310089    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26" podUID="17767cc6-6a5e-4bb9-84d2-98162742cd43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 31 13:42:33.626794 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:33.626754    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"]
Oct 31 13:42:33.627575 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:42:33.627542    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5fe2659_7755_4acc_adcd_481f1d5fa982.slice/crio-f88133e19a59bb62c142d276789c6871ed3c834cae0396098aa372bd53f641a1 WatchSource:0}: Error finding container f88133e19a59bb62c142d276789c6871ed3c834cae0396098aa372bd53f641a1: Status 404 returned error can't find the container with id f88133e19a59bb62c142d276789c6871ed3c834cae0396098aa372bd53f641a1
Oct 31 13:42:34.315116 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:34.315087    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:42:34.315643 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:34.315572    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:42:34.316385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:34.316363    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:42:34.385547 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:34.385502    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g" event={"ID":"a5fe2659-7755-4acc-adcd-481f1d5fa982","Type":"ContainerStarted","Data":"68d4fb0691c329e7d4142f1e4078a2061ed202264117ed48544e8bf0690e6d7e"}
Oct 31 13:42:34.385753 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:34.385566    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g" event={"ID":"a5fe2659-7755-4acc-adcd-481f1d5fa982","Type":"ContainerStarted","Data":"f88133e19a59bb62c142d276789c6871ed3c834cae0396098aa372bd53f641a1"}
Oct 31 13:42:34.988553 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:34.988515    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:42.059639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:42.059580    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g" podStartSLOduration=10.059564329 podStartE2EDuration="10.059564329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:42:34.41554897 +0000 UTC m=+2354.034208134" watchObservedRunningTime="2026-10-31 13:42:42.059564329 +0000 UTC m=+2361.678223495"
Oct 31 13:42:42.060471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:42.060449    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"]
Oct 31 13:42:42.060682 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:42.060659    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g" podUID="1fc4155a-edd2-4907-a3c1-741f6ecefcdb" containerName="istio-proxy" containerID="cri-o://b79a596ebcc20a2628bd7ee4c7474d1c523b98daefbbfbe6939d199ab808582f" gracePeriod=30
Oct 31 13:42:42.987382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:42.987348    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:43.000463 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:43.000439    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:43.499175 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:43.499140    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:47.442331 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.441792    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:42:47.485681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.485645    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"]
Oct 31 13:42:47.486911 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.486887    4660 cpu_manager.go:414] "containerMap: removing container" podUID="1fc4155a-edd2-4907-a3c1-741f6ecefcdb" containerName="istio-proxy"
Oct 31 13:42:47.486980 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.486916    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc4155a-edd2-4907-a3c1-741f6ecefcdb" containerName="istio-proxy"
Oct 31 13:42:47.487237 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.487219    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc4155a-edd2-4907-a3c1-741f6ecefcdb" containerName="istio-proxy"
Oct 31 13:42:47.492595 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.492568    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.498000 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.497974    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fb50fa-openshift-default-dockercfg-7tzdl\""
Oct 31 13:42:47.519819 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.519786    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"]
Oct 31 13:42:47.537562 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.537529    4660 generic.go:356] "Generic (PLEG): container finished" podID="1fc4155a-edd2-4907-a3c1-741f6ecefcdb" containerID="b79a596ebcc20a2628bd7ee4c7474d1c523b98daefbbfbe6939d199ab808582f" exitCode=0
Oct 31 13:42:47.538047 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.538025    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"
Oct 31 13:42:47.538344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.538282    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g" event={"ID":"1fc4155a-edd2-4907-a3c1-741f6ecefcdb","Type":"ContainerDied","Data":"b79a596ebcc20a2628bd7ee4c7474d1c523b98daefbbfbe6939d199ab808582f"}
Oct 31 13:42:47.538440 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.538346    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g" event={"ID":"1fc4155a-edd2-4907-a3c1-741f6ecefcdb","Type":"ContainerDied","Data":"aeeee131c43d374c4009ebdde5c39022041a7fb3ad7e382ec4180d9ac7f199ae"}
Oct 31 13:42:47.538440 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.538376    4660 scope.go:122] "RemoveContainer" containerID="b79a596ebcc20a2628bd7ee4c7474d1c523b98daefbbfbe6939d199ab808582f"
Oct 31 13:42:47.570236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.570130    4660 scope.go:122] "RemoveContainer" containerID="b79a596ebcc20a2628bd7ee4c7474d1c523b98daefbbfbe6939d199ab808582f"
Oct 31 13:42:47.583454 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:42:47.583401    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b79a596ebcc20a2628bd7ee4c7474d1c523b98daefbbfbe6939d199ab808582f\": container with ID starting with b79a596ebcc20a2628bd7ee4c7474d1c523b98daefbbfbe6939d199ab808582f not found: ID does not exist" containerID="b79a596ebcc20a2628bd7ee4c7474d1c523b98daefbbfbe6939d199ab808582f"
Oct 31 13:42:47.583597 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.583482    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b79a596ebcc20a2628bd7ee4c7474d1c523b98daefbbfbe6939d199ab808582f"} err="failed to get container status \"b79a596ebcc20a2628bd7ee4c7474d1c523b98daefbbfbe6939d199ab808582f\": rpc error: code = NotFound desc = could not find container \"b79a596ebcc20a2628bd7ee4c7474d1c523b98daefbbfbe6939d199ab808582f\": container with ID starting with b79a596ebcc20a2628bd7ee4c7474d1c523b98daefbbfbe6939d199ab808582f not found: ID does not exist"
Oct 31 13:42:47.591074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.591046    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-podinfo\") pod \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") "
Oct 31 13:42:47.591747 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.591091    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-workload-socket\") pod \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") "
Oct 31 13:42:47.591747 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.591136    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-credential-socket\") pod \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") "
Oct 31 13:42:47.591747 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.591341    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-kube-api-access-cbxcg\" (UniqueName: \"kubernetes.io/projected/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-kube-api-access-cbxcg\") pod \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") "
Oct 31 13:42:47.591747 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.591384    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istiod-ca-cert\") pod \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") "
Oct 31 13:42:47.591747 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.591487    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-workload-socket" pod "1fc4155a-edd2-4907-a3c1-741f6ecefcdb" (UID: "1fc4155a-edd2-4907-a3c1-741f6ecefcdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:42:47.591747 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.591498    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-envoy\") pod \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") "
Oct 31 13:42:47.591747 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.591585    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-data\") pod \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") "
Oct 31 13:42:47.591747 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.591675    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-workload-certs\") pod \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") "
Oct 31 13:42:47.592098 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.591774    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-token\" (UniqueName: \"kubernetes.io/projected/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-token\") pod \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\" (UID: \"1fc4155a-edd2-4907-a3c1-741f6ecefcdb\") "
Oct 31 13:42:47.592173 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.592154    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-credential-socket\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.592343 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.592325    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istiod-ca-cert\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.592450 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.592432    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx57f\" (UniqueName: \"kubernetes.io/projected/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-kube-api-access-fx57f\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.592829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.592605    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-workload-socket\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.592829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.592616    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-credential-socket" pod "1fc4155a-edd2-4907-a3c1-741f6ecefcdb" (UID: "1fc4155a-edd2-4907-a3c1-741f6ecefcdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:42:47.592829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.592697    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-envoy\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.592829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.592734    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-data" pod "1fc4155a-edd2-4907-a3c1-741f6ecefcdb" (UID: "1fc4155a-edd2-4907-a3c1-741f6ecefcdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:42:47.592829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.592755    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istiod-ca-cert" pod "1fc4155a-edd2-4907-a3c1-741f6ecefcdb" (UID: "1fc4155a-edd2-4907-a3c1-741f6ecefcdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:42:47.592829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.592738    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-podinfo\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.592829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.592823    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-workload-certs" pod "1fc4155a-edd2-4907-a3c1-741f6ecefcdb" (UID: "1fc4155a-edd2-4907-a3c1-741f6ecefcdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:42:47.595225 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.592914    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-workload-certs\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.595225 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.592940    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-data\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.595225 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.592989    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-token\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.595225 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.593121    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:42:47.595225 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.593149    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:42:47.595225 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.593167    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:42:47.595225 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.593182    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:42:47.595225 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.593197    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:42:47.599261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.599148    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-token" pod "1fc4155a-edd2-4907-a3c1-741f6ecefcdb" (UID: "1fc4155a-edd2-4907-a3c1-741f6ecefcdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:42:47.600105 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.600075    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-podinfo" pod "1fc4155a-edd2-4907-a3c1-741f6ecefcdb" (UID: "1fc4155a-edd2-4907-a3c1-741f6ecefcdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:42:47.600201 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.600118    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-kube-api-access-cbxcg" pod "1fc4155a-edd2-4907-a3c1-741f6ecefcdb" (UID: "1fc4155a-edd2-4907-a3c1-741f6ecefcdb"). InnerVolumeSpecName "kube-api-access-cbxcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:42:47.603613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.603588    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-envoy" pod "1fc4155a-edd2-4907-a3c1-741f6ecefcdb" (UID: "1fc4155a-edd2-4907-a3c1-741f6ecefcdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:42:47.694828 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.694721    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx57f\" (UniqueName: \"kubernetes.io/projected/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-kube-api-access-fx57f\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.695022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.694882    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-workload-socket\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.695022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.694914    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-envoy\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.695022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.694960    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-podinfo\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.695022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.695000    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-workload-certs\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.697471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.695027    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-data\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.697471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.695089    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-token\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.697471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.695152    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-credential-socket\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.697471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.695340    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istiod-ca-cert\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.697471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.695797    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbxcg\" (UniqueName: \"kubernetes.io/projected/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-kube-api-access-cbxcg\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:42:47.697471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.696344    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-workload-socket\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.697471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.696414    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istiod-ca-cert\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.697471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.696718    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-data\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.697471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.696806    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-credential-socket\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.697471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.697089    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:42:47.697471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.697116    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:42:47.697471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.697132    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc4155a-edd2-4907-a3c1-741f6ecefcdb-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:42:47.698256 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.697838    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-workload-certs\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.698359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.698339    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-envoy\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.700142 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.700117    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-podinfo\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.708959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.708933    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx57f\" (UniqueName: \"kubernetes.io/projected/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-kube-api-access-fx57f\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.711040 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.711013    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-token\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.814858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.814821    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:47.874374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.874340    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"]
Oct 31 13:42:47.879810 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:47.879784    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sxr7g"]
Oct 31 13:42:48.125468 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:42:48.125428    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf482cd7_2dd2_4cb0_b791_e4a9f8c1e097.slice/crio-a76df8b6f7bd182a57d30bef9aee2192bbb02356bd01373ee6e24ecc4582e662 WatchSource:0}: Error finding container a76df8b6f7bd182a57d30bef9aee2192bbb02356bd01373ee6e24ecc4582e662: Status 404 returned error can't find the container with id a76df8b6f7bd182a57d30bef9aee2192bbb02356bd01373ee6e24ecc4582e662
Oct 31 13:42:48.129208 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:48.129181    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"]
Oct 31 13:42:48.129468 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:48.129430    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:42:48.129535 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:48.129518    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:42:48.129576 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:48.129564    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:42:48.535936 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:48.535854    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fc4155a-edd2-4907-a3c1-741f6ecefcdb" path="/var/lib/kubelet/pods/1fc4155a-edd2-4907-a3c1-741f6ecefcdb/volumes"
Oct 31 13:42:48.553601 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:48.553571    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b" event={"ID":"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097","Type":"ContainerStarted","Data":"a407c755e738f29adca841b5c8a767ec8f9b6a43416d8da4c875f0bdacc6d36a"}
Oct 31 13:42:48.553742 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:48.553626    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b" event={"ID":"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097","Type":"ContainerStarted","Data":"a76df8b6f7bd182a57d30bef9aee2192bbb02356bd01373ee6e24ecc4582e662"}
Oct 31 13:42:48.583659 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:48.583599    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b" podStartSLOduration=14.583582021 podStartE2EDuration="14.583582021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:42:48.578815159 +0000 UTC m=+2368.197474336" watchObservedRunningTime="2026-10-31 13:42:48.583582021 +0000 UTC m=+2368.202241186"
Oct 31 13:42:48.815155 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:48.815057    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:49.830446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:49.830405    4660 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 13:42:49.831028 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:49.830992    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b" podUID="df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 13:42:50.768003 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:42:50.767967    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:42:50.816828 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:50.816788    4660 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 13:42:50.817111 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:50.817082    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b" podUID="df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 13:42:51.823777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:51.823736    4660 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 13:42:51.824411 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:51.824376    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b" podUID="df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 13:42:52.816315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:52.815995    4660 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 13:42:52.816315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:52.816060    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b" podUID="df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 13:42:53.818140 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:53.818104    4660 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 13:42:53.845328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:53.831157    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b" podUID="df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 13:42:54.233414 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:54.228447    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49" podUID="982ab639-4caa-445f-b029-4390d81cfad5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.131:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:42:54.387162 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:54.386401    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j" podUID="740f8e32-74e2-4c64-8d14-e0124a90bd5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:42:54.493365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:54.492242    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g" podUID="a5fe2659-7755-4acc-adcd-481f1d5fa982" containerName="main" probeResult="failure" output="Get \"https://10.133.0.148:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:42:54.669322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:54.669196    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:42:54.704793 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:54.702390    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4" podUID="708ebdbd-2c8c-4d47-8d03-517e149613d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.144:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:42:54.898314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:54.897709    4660 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 13:42:54.898314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:54.897769    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b" podUID="df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 13:42:54.938476 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:54.938090    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:42:55.140694 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.130282    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584" podUID="59caaca5-1f5a-42fc-841d-234c76fef93e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.147:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:42:55.140694 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.130641    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 31 13:42:55.140694 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.130873    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz" podUID="f3e60e1e-371d-4a37-b988-e76510d17e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": context deadline exceeded"
Oct 31 13:42:55.145470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.145420    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:42:55.148393 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.148282    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:42:55.156514 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.156483    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm" podUID="cdc22df5-69c4-4def-9e7d-f022057cd410" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:42:55.160200 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.160165    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:42:55.235317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.231953    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49" podUID="982ab639-4caa-445f-b029-4390d81cfad5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:42:55.235317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.232378    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq" podUID="13349545-8422-4466-9152-bd143810b334" containerName="main" probeResult="failure" output="Get \"https://10.133.0.141:8000/ready\": context deadline exceeded"
Oct 31 13:42:55.235317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.233110    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:42:55.247498 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.247437    4660 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4" podUID="708ebdbd-2c8c-4d47-8d03-517e149613d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.144:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:42:55.495046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.494955    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g" podUID="a5fe2659-7755-4acc-adcd-481f1d5fa982" containerName="main" probeResult="failure" output="Get \"https://10.133.0.148:8000/ready\": context deadline exceeded"
Oct 31 13:42:55.496433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.496411    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:55.669391 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.669342    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:42:55.671052 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.670927    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:42:55.722328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.716015    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 13:42:55.781765 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.781682    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:42:55.799578 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.796449    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:42:55.833318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.817821    4660 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 13:42:55.833318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:55.817900    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b" podUID="df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 13:42:56.979577 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:56.979543    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:56.980545 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:56.980516    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:42:56.981352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:42:56.981332    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:43:20.735512 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:43:20.735426    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:43:20.840022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:20.839983    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"]
Oct 31 13:43:20.845512 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:20.845486    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"
Oct 31 13:43:20.851460 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:20.850851    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:43:20.851460 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:20.850989    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:43:20.853710 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:20.853680    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"]
Oct 31 13:43:20.966527 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:20.966491    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9eb56c2-99tnf\" (UID: \"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"
Oct 31 13:43:20.966700 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:20.966636    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rljt8\" (UniqueName: \"kubernetes.io/projected/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-kube-api-access-rljt8\") pod \"maas-api-revoke-sub-d9eb56c2-99tnf\" (UID: \"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"
Oct 31 13:43:20.966920 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:20.966900    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9eb56c2-99tnf\" (UID: \"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"
Oct 31 13:43:21.041729 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.041641    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:43:21.042254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.042231    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:43:21.049586 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.049565    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:43:21.049899 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.049879    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:43:21.063019 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.062997    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:43:21.063198 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.063179    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:43:21.068348 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.068321    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9eb56c2-99tnf\" (UID: \"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"
Oct 31 13:43:21.068472 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.068453    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rljt8\" (UniqueName: \"kubernetes.io/projected/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-kube-api-access-rljt8\") pod \"maas-api-revoke-sub-d9eb56c2-99tnf\" (UID: \"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"
Oct 31 13:43:21.068750 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.068718    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9eb56c2-99tnf\" (UID: \"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"
Oct 31 13:43:21.069463 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.069440    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9eb56c2-99tnf\" (UID: \"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"
Oct 31 13:43:21.071141 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.071121    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:43:21.081983 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.081961    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9eb56c2-99tnf\" (UID: \"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"
Oct 31 13:43:21.086937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.086913    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rljt8\" (UniqueName: \"kubernetes.io/projected/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-kube-api-access-rljt8\") pod \"maas-api-revoke-sub-d9eb56c2-99tnf\" (UID: \"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"
Oct 31 13:43:21.171872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.171845    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:43:21.179001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.178976    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"
Oct 31 13:43:21.497754 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:43:21.497725    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2bc921_7c70_4c8d_8031_ca5462e0c2dc.slice/crio-6e013ef124403e4b12028eb9e1176cf27002c7da8dccb3b1be20a44a8609a361 WatchSource:0}: Error finding container 6e013ef124403e4b12028eb9e1176cf27002c7da8dccb3b1be20a44a8609a361: Status 404 returned error can't find the container with id 6e013ef124403e4b12028eb9e1176cf27002c7da8dccb3b1be20a44a8609a361
Oct 31 13:43:21.505521 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.505502    4660 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:43:21.509112 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.509087    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"]
Oct 31 13:43:21.682437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:21.682384    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf" event={"ID":"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc","Type":"ContainerStarted","Data":"6e013ef124403e4b12028eb9e1176cf27002c7da8dccb3b1be20a44a8609a361"}
Oct 31 13:43:25.781951 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:25.781911    4660 generic.go:356] "Generic (PLEG): container finished" podID="ec2bc921-7c70-4c8d-8031-ca5462e0c2dc" containerID="061f26d8b70298ce48c23c493c48dec24510427fedecdc0c987c5fda2ae1a567" exitCode=0
Oct 31 13:43:25.782417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:25.781981    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf" event={"ID":"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc","Type":"ContainerDied","Data":"061f26d8b70298ce48c23c493c48dec24510427fedecdc0c987c5fda2ae1a567"}
Oct 31 13:43:26.992507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:26.992482    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"
Oct 31 13:43:27.039722 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:27.039692    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-maas-api-openshift-ca\") pod \"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc\" (UID: \"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc\") "
Oct 31 13:43:27.039977 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:27.039959    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-kube-api-access-rljt8\" (UniqueName: \"kubernetes.io/projected/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-kube-api-access-rljt8\") pod \"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc\" (UID: \"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc\") "
Oct 31 13:43:27.040200 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:27.040178    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-maas-api-xks-ca\") pod \"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc\" (UID: \"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc\") "
Oct 31 13:43:27.041268 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:27.040391    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-maas-api-openshift-ca" pod "ec2bc921-7c70-4c8d-8031-ca5462e0c2dc" (UID: "ec2bc921-7c70-4c8d-8031-ca5462e0c2dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:43:27.042165 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:27.042110    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-maas-api-openshift-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:43:27.045003 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:27.044897    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-kube-api-access-rljt8" pod "ec2bc921-7c70-4c8d-8031-ca5462e0c2dc" (UID: "ec2bc921-7c70-4c8d-8031-ca5462e0c2dc"). InnerVolumeSpecName "kube-api-access-rljt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:43:27.047080 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:27.047053    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-maas-api-xks-ca" pod "ec2bc921-7c70-4c8d-8031-ca5462e0c2dc" (UID: "ec2bc921-7c70-4c8d-8031-ca5462e0c2dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:43:27.144505 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:27.144427    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rljt8\" (UniqueName: \"kubernetes.io/projected/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-kube-api-access-rljt8\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:43:27.144505 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:27.144477    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc-maas-api-xks-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:43:27.801461 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:27.801434    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"
Oct 31 13:43:27.801649 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:27.801471    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf" event={"ID":"ec2bc921-7c70-4c8d-8031-ca5462e0c2dc","Type":"ContainerDied","Data":"6e013ef124403e4b12028eb9e1176cf27002c7da8dccb3b1be20a44a8609a361"}
Oct 31 13:43:27.801649 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:27.801530    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e013ef124403e4b12028eb9e1176cf27002c7da8dccb3b1be20a44a8609a361"
Oct 31 13:43:31.565321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:31.563030    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"]
Oct 31 13:43:32.132325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.132263    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"]
Oct 31 13:43:32.133601 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.133572    4660 cpu_manager.go:414] "containerMap: removing container" podUID="ec2bc921-7c70-4c8d-8031-ca5462e0c2dc" containerName="revoke-keys"
Oct 31 13:43:32.133727 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.133601    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2bc921-7c70-4c8d-8031-ca5462e0c2dc" containerName="revoke-keys"
Oct 31 13:43:32.133852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.133839    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2bc921-7c70-4c8d-8031-ca5462e0c2dc" containerName="revoke-keys"
Oct 31 13:43:32.138562 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.138538    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"
Oct 31 13:43:32.148959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.148934    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:43:32.150804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.149608    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:43:32.167499 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.167468    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"]
Oct 31 13:43:32.275654 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.275618    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7cd289-424c-434f-981b-d83931c4feb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15ce5bb1-t2l9l\" (UID: \"5e7cd289-424c-434f-981b-d83931c4feb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"
Oct 31 13:43:32.275931 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.275908    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hmbc\" (UniqueName: \"kubernetes.io/projected/5e7cd289-424c-434f-981b-d83931c4feb7-kube-api-access-2hmbc\") pod \"maas-api-revoke-sub-15ce5bb1-t2l9l\" (UID: \"5e7cd289-424c-434f-981b-d83931c4feb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"
Oct 31 13:43:32.276115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.276093    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7cd289-424c-434f-981b-d83931c4feb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15ce5bb1-t2l9l\" (UID: \"5e7cd289-424c-434f-981b-d83931c4feb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"
Oct 31 13:43:32.377842 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.377806    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hmbc\" (UniqueName: \"kubernetes.io/projected/5e7cd289-424c-434f-981b-d83931c4feb7-kube-api-access-2hmbc\") pod \"maas-api-revoke-sub-15ce5bb1-t2l9l\" (UID: \"5e7cd289-424c-434f-981b-d83931c4feb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"
Oct 31 13:43:32.378038 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.377948    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7cd289-424c-434f-981b-d83931c4feb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15ce5bb1-t2l9l\" (UID: \"5e7cd289-424c-434f-981b-d83931c4feb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"
Oct 31 13:43:32.378106 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.378060    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7cd289-424c-434f-981b-d83931c4feb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15ce5bb1-t2l9l\" (UID: \"5e7cd289-424c-434f-981b-d83931c4feb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"
Oct 31 13:43:32.378617 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.378596    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7cd289-424c-434f-981b-d83931c4feb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15ce5bb1-t2l9l\" (UID: \"5e7cd289-424c-434f-981b-d83931c4feb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"
Oct 31 13:43:32.380538 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.380517    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7cd289-424c-434f-981b-d83931c4feb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15ce5bb1-t2l9l\" (UID: \"5e7cd289-424c-434f-981b-d83931c4feb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"
Oct 31 13:43:32.391786 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.391732    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hmbc\" (UniqueName: \"kubernetes.io/projected/5e7cd289-424c-434f-981b-d83931c4feb7-kube-api-access-2hmbc\") pod \"maas-api-revoke-sub-15ce5bb1-t2l9l\" (UID: \"5e7cd289-424c-434f-981b-d83931c4feb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"
Oct 31 13:43:32.465618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.465577    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"
Oct 31 13:43:32.781436 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:43:32.781381    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7cd289_424c_434f_981b_d83931c4feb7.slice/crio-8ed7868665f01c903f94f32111b67cb988391a5d90cd58d89de24a7763443370 WatchSource:0}: Error finding container 8ed7868665f01c903f94f32111b67cb988391a5d90cd58d89de24a7763443370: Status 404 returned error can't find the container with id 8ed7868665f01c903f94f32111b67cb988391a5d90cd58d89de24a7763443370
Oct 31 13:43:32.785418 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.785332    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"]
Oct 31 13:43:32.919071 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:32.919020    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l" event={"ID":"5e7cd289-424c-434f-981b-d83931c4feb7","Type":"ContainerStarted","Data":"8ed7868665f01c903f94f32111b67cb988391a5d90cd58d89de24a7763443370"}
Oct 31 13:43:33.934889 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:33.934849    4660 generic.go:356] "Generic (PLEG): container finished" podID="5e7cd289-424c-434f-981b-d83931c4feb7" containerID="e4d56369bddf62db87f48b507f8c18d099e1cbf263c7459d70ff093b839e4c01" exitCode=0
Oct 31 13:43:33.935320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:33.934926    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l" event={"ID":"5e7cd289-424c-434f-981b-d83931c4feb7","Type":"ContainerDied","Data":"e4d56369bddf62db87f48b507f8c18d099e1cbf263c7459d70ff093b839e4c01"}
Oct 31 13:43:35.173730 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:35.173704    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"
Oct 31 13:43:35.358098 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:35.358009    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e7cd289-424c-434f-981b-d83931c4feb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7cd289-424c-434f-981b-d83931c4feb7-maas-api-openshift-ca\") pod \"5e7cd289-424c-434f-981b-d83931c4feb7\" (UID: \"5e7cd289-424c-434f-981b-d83931c4feb7\") "
Oct 31 13:43:35.358290 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:35.358268    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e7cd289-424c-434f-981b-d83931c4feb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7cd289-424c-434f-981b-d83931c4feb7-maas-api-xks-ca\") pod \"5e7cd289-424c-434f-981b-d83931c4feb7\" (UID: \"5e7cd289-424c-434f-981b-d83931c4feb7\") "
Oct 31 13:43:35.358410 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:35.358391    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e7cd289-424c-434f-981b-d83931c4feb7-kube-api-access-2hmbc\" (UniqueName: \"kubernetes.io/projected/5e7cd289-424c-434f-981b-d83931c4feb7-kube-api-access-2hmbc\") pod \"5e7cd289-424c-434f-981b-d83931c4feb7\" (UID: \"5e7cd289-424c-434f-981b-d83931c4feb7\") "
Oct 31 13:43:35.358789 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:35.358637    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e7cd289-424c-434f-981b-d83931c4feb7-maas-api-openshift-ca" pod "5e7cd289-424c-434f-981b-d83931c4feb7" (UID: "5e7cd289-424c-434f-981b-d83931c4feb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:43:35.359242 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:35.359213    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7cd289-424c-434f-981b-d83931c4feb7-maas-api-openshift-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:43:35.360998 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:35.360970    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7cd289-424c-434f-981b-d83931c4feb7-kube-api-access-2hmbc" pod "5e7cd289-424c-434f-981b-d83931c4feb7" (UID: "5e7cd289-424c-434f-981b-d83931c4feb7"). InnerVolumeSpecName "kube-api-access-2hmbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:43:35.361112 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:35.360988    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e7cd289-424c-434f-981b-d83931c4feb7-maas-api-xks-ca" pod "5e7cd289-424c-434f-981b-d83931c4feb7" (UID: "5e7cd289-424c-434f-981b-d83931c4feb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:43:35.461123 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:35.461078    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7cd289-424c-434f-981b-d83931c4feb7-maas-api-xks-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:43:35.461123 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:35.461114    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hmbc\" (UniqueName: \"kubernetes.io/projected/5e7cd289-424c-434f-981b-d83931c4feb7-kube-api-access-2hmbc\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:43:35.957861 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:35.957832    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"
Oct 31 13:43:35.958041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:35.957867    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l" event={"ID":"5e7cd289-424c-434f-981b-d83931c4feb7","Type":"ContainerDied","Data":"8ed7868665f01c903f94f32111b67cb988391a5d90cd58d89de24a7763443370"}
Oct 31 13:43:35.958041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:35.957916    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ed7868665f01c903f94f32111b67cb988391a5d90cd58d89de24a7763443370"
Oct 31 13:43:46.617069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:46.617004    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49" podUID="982ab639-4caa-445f-b029-4390d81cfad5" containerName="main" containerID="cri-o://9b1f56d1d839b787f5a7549bdc7a876f2fc67b9e2f3355680f5dd94f42dd3bd5" gracePeriod=45
Oct 31 13:43:46.740323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:46.740277    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49" podUID="982ab639-4caa-445f-b029-4390d81cfad5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.131:8000/ready\": dial tcp 10.133.0.131:8000: connect: connection refused"
Oct 31 13:43:46.982964 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:46.982931    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:43:47.040566 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.040524    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-home\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-home\") pod \"982ab639-4caa-445f-b029-4390d81cfad5\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") "
Oct 31 13:43:47.040566 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.040569    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-tmp-dir\") pod \"982ab639-4caa-445f-b029-4390d81cfad5\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") "
Oct 31 13:43:47.040822 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.040615    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-dshm\") pod \"982ab639-4caa-445f-b029-4390d81cfad5\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") "
Oct 31 13:43:47.040822 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.040668    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982ab639-4caa-445f-b029-4390d81cfad5-kube-api-access-97l7c\" (UniqueName: \"kubernetes.io/projected/982ab639-4caa-445f-b029-4390d81cfad5-kube-api-access-97l7c\") pod \"982ab639-4caa-445f-b029-4390d81cfad5\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") "
Oct 31 13:43:47.040822 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.040699    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-model-cache\") pod \"982ab639-4caa-445f-b029-4390d81cfad5\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") "
Oct 31 13:43:47.040997 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.040981    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/982ab639-4caa-445f-b029-4390d81cfad5-tls-certs\" (UniqueName: \"kubernetes.io/secret/982ab639-4caa-445f-b029-4390d81cfad5-tls-certs\") pod \"982ab639-4caa-445f-b029-4390d81cfad5\" (UID: \"982ab639-4caa-445f-b029-4390d81cfad5\") "
Oct 31 13:43:47.047954 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.044560    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982ab639-4caa-445f-b029-4390d81cfad5-tls-certs" pod "982ab639-4caa-445f-b029-4390d81cfad5" (UID: "982ab639-4caa-445f-b029-4390d81cfad5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:43:47.047954 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.045606    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-home" pod "982ab639-4caa-445f-b029-4390d81cfad5" (UID: "982ab639-4caa-445f-b029-4390d81cfad5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:43:47.047954 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.045840    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-tmp-dir" pod "982ab639-4caa-445f-b029-4390d81cfad5" (UID: "982ab639-4caa-445f-b029-4390d81cfad5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:43:47.051360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.051283    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-model-cache" pod "982ab639-4caa-445f-b029-4390d81cfad5" (UID: "982ab639-4caa-445f-b029-4390d81cfad5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:43:47.052456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.052426    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-dshm" pod "982ab639-4caa-445f-b029-4390d81cfad5" (UID: "982ab639-4caa-445f-b029-4390d81cfad5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:43:47.053440 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.053414    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982ab639-4caa-445f-b029-4390d81cfad5-kube-api-access-97l7c" pod "982ab639-4caa-445f-b029-4390d81cfad5" (UID: "982ab639-4caa-445f-b029-4390d81cfad5"). InnerVolumeSpecName "kube-api-access-97l7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:43:47.107694 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.107661    4660 generic.go:356] "Generic (PLEG): container finished" podID="982ab639-4caa-445f-b029-4390d81cfad5" containerID="9b1f56d1d839b787f5a7549bdc7a876f2fc67b9e2f3355680f5dd94f42dd3bd5" exitCode=0
Oct 31 13:43:47.107873 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.107719    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"
Oct 31 13:43:47.107873 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.107737    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49" event={"ID":"982ab639-4caa-445f-b029-4390d81cfad5","Type":"ContainerDied","Data":"9b1f56d1d839b787f5a7549bdc7a876f2fc67b9e2f3355680f5dd94f42dd3bd5"}
Oct 31 13:43:47.107873 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.107785    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49" event={"ID":"982ab639-4caa-445f-b029-4390d81cfad5","Type":"ContainerDied","Data":"e457914ac333e0bd7b9efc3d97c4a012ca51357a4bdabff170e49fca039cd1e7"}
Oct 31 13:43:47.107873 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.107808    4660 scope.go:122] "RemoveContainer" containerID="9b1f56d1d839b787f5a7549bdc7a876f2fc67b9e2f3355680f5dd94f42dd3bd5"
Oct 31 13:43:47.127005 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.126947    4660 scope.go:122] "RemoveContainer" containerID="9b1f56d1d839b787f5a7549bdc7a876f2fc67b9e2f3355680f5dd94f42dd3bd5"
Oct 31 13:43:47.127442 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:43:47.127353    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b1f56d1d839b787f5a7549bdc7a876f2fc67b9e2f3355680f5dd94f42dd3bd5\": container with ID starting with 9b1f56d1d839b787f5a7549bdc7a876f2fc67b9e2f3355680f5dd94f42dd3bd5 not found: ID does not exist" containerID="9b1f56d1d839b787f5a7549bdc7a876f2fc67b9e2f3355680f5dd94f42dd3bd5"
Oct 31 13:43:47.127442 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.127397    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b1f56d1d839b787f5a7549bdc7a876f2fc67b9e2f3355680f5dd94f42dd3bd5"} err="failed to get container status \"9b1f56d1d839b787f5a7549bdc7a876f2fc67b9e2f3355680f5dd94f42dd3bd5\": rpc error: code = NotFound desc = could not find container \"9b1f56d1d839b787f5a7549bdc7a876f2fc67b9e2f3355680f5dd94f42dd3bd5\": container with ID starting with 9b1f56d1d839b787f5a7549bdc7a876f2fc67b9e2f3355680f5dd94f42dd3bd5 not found: ID does not exist"
Oct 31 13:43:47.143289 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.143260    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982ab639-4caa-445f-b029-4390d81cfad5-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:43:47.143289 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.143288    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:43:47.143289 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.143316    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:43:47.143289 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.143327    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:43:47.143626 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.143341    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97l7c\" (UniqueName: \"kubernetes.io/projected/982ab639-4caa-445f-b029-4390d81cfad5-kube-api-access-97l7c\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:43:47.143626 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.143356    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982ab639-4caa-445f-b029-4390d81cfad5-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:43:47.167930 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.167868    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"]
Oct 31 13:43:47.174227 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:47.174199    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-69bdf5/test-model-69bdf5-kserve-b6566b447-vtv49"]
Oct 31 13:43:48.535219 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:48.535185    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982ab639-4caa-445f-b029-4390d81cfad5" path="/var/lib/kubelet/pods/982ab639-4caa-445f-b029-4390d81cfad5/volumes"
Oct 31 13:43:48.902008 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:48.901977    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"]
Oct 31 13:43:49.385375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.377272    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"]
Oct 31 13:43:49.385375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.379653    4660 cpu_manager.go:414] "containerMap: removing container" podUID="982ab639-4caa-445f-b029-4390d81cfad5" containerName="main"
Oct 31 13:43:49.385375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.379681    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982ab639-4caa-445f-b029-4390d81cfad5" containerName="main"
Oct 31 13:43:49.385375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.379843    4660 cpu_manager.go:414] "containerMap: removing container" podUID="5e7cd289-424c-434f-981b-d83931c4feb7" containerName="revoke-keys"
Oct 31 13:43:49.385375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.379852    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e7cd289-424c-434f-981b-d83931c4feb7" containerName="revoke-keys"
Oct 31 13:43:49.385375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.380347    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e7cd289-424c-434f-981b-d83931c4feb7" containerName="revoke-keys"
Oct 31 13:43:49.385375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.380380    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="982ab639-4caa-445f-b029-4390d81cfad5" containerName="main"
Oct 31 13:43:49.438245 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.438216    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"]
Oct 31 13:43:49.438399 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.438358    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"
Oct 31 13:43:49.446467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.445962    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:43:49.446467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.446248    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:43:49.584765 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.584709    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d374cdd2-4367-4745-8290-33a05145a48d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2\" (UID: \"d374cdd2-4367-4745-8290-33a05145a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"
Oct 31 13:43:49.585235 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.584952    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v67vc\" (UniqueName: \"kubernetes.io/projected/d374cdd2-4367-4745-8290-33a05145a48d-kube-api-access-v67vc\") pod \"maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2\" (UID: \"d374cdd2-4367-4745-8290-33a05145a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"
Oct 31 13:43:49.585315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.585225    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d374cdd2-4367-4745-8290-33a05145a48d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2\" (UID: \"d374cdd2-4367-4745-8290-33a05145a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"
Oct 31 13:43:49.687015 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.686941    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d374cdd2-4367-4745-8290-33a05145a48d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2\" (UID: \"d374cdd2-4367-4745-8290-33a05145a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"
Oct 31 13:43:49.687157 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.687063    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v67vc\" (UniqueName: \"kubernetes.io/projected/d374cdd2-4367-4745-8290-33a05145a48d-kube-api-access-v67vc\") pod \"maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2\" (UID: \"d374cdd2-4367-4745-8290-33a05145a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"
Oct 31 13:43:49.687234 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.687218    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d374cdd2-4367-4745-8290-33a05145a48d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2\" (UID: \"d374cdd2-4367-4745-8290-33a05145a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"
Oct 31 13:43:49.687969 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.687935    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d374cdd2-4367-4745-8290-33a05145a48d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2\" (UID: \"d374cdd2-4367-4745-8290-33a05145a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"
Oct 31 13:43:49.689800 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.689773    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d374cdd2-4367-4745-8290-33a05145a48d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2\" (UID: \"d374cdd2-4367-4745-8290-33a05145a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"
Oct 31 13:43:49.713245 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.713218    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v67vc\" (UniqueName: \"kubernetes.io/projected/d374cdd2-4367-4745-8290-33a05145a48d-kube-api-access-v67vc\") pod \"maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2\" (UID: \"d374cdd2-4367-4745-8290-33a05145a48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"
Oct 31 13:43:49.762783 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:49.762750    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"
Oct 31 13:43:50.075216 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:50.075181    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"]
Oct 31 13:43:50.446158 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:50.446119    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2" event={"ID":"d374cdd2-4367-4745-8290-33a05145a48d","Type":"ContainerStarted","Data":"02764320ea1a8c9b3eb65bd4a4259624fce75128885fc49e5b7f7a115731f653"}
Oct 31 13:43:50.738474 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:43:50.738383    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:43:51.464267 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:51.464230    4660 generic.go:356] "Generic (PLEG): container finished" podID="d374cdd2-4367-4745-8290-33a05145a48d" containerID="b78b25373dda7a5943a532459b5d969b2f63f12aaf5d6495d195005a04265d58" exitCode=0
Oct 31 13:43:51.464490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:51.464330    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2" event={"ID":"d374cdd2-4367-4745-8290-33a05145a48d","Type":"ContainerDied","Data":"b78b25373dda7a5943a532459b5d969b2f63f12aaf5d6495d195005a04265d58"}
Oct 31 13:43:52.674666 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:52.674643    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"
Oct 31 13:43:52.761105 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:52.761008    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d374cdd2-4367-4745-8290-33a05145a48d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d374cdd2-4367-4745-8290-33a05145a48d-maas-api-openshift-ca\") pod \"d374cdd2-4367-4745-8290-33a05145a48d\" (UID: \"d374cdd2-4367-4745-8290-33a05145a48d\") "
Oct 31 13:43:52.761105 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:52.761093    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d374cdd2-4367-4745-8290-33a05145a48d-kube-api-access-v67vc\" (UniqueName: \"kubernetes.io/projected/d374cdd2-4367-4745-8290-33a05145a48d-kube-api-access-v67vc\") pod \"d374cdd2-4367-4745-8290-33a05145a48d\" (UID: \"d374cdd2-4367-4745-8290-33a05145a48d\") "
Oct 31 13:43:52.762070 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:52.761413    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d374cdd2-4367-4745-8290-33a05145a48d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d374cdd2-4367-4745-8290-33a05145a48d-maas-api-xks-ca\") pod \"d374cdd2-4367-4745-8290-33a05145a48d\" (UID: \"d374cdd2-4367-4745-8290-33a05145a48d\") "
Oct 31 13:43:52.762070 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:52.761441    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d374cdd2-4367-4745-8290-33a05145a48d-maas-api-openshift-ca" pod "d374cdd2-4367-4745-8290-33a05145a48d" (UID: "d374cdd2-4367-4745-8290-33a05145a48d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:43:52.762435 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:52.762408    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d374cdd2-4367-4745-8290-33a05145a48d-maas-api-openshift-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:43:52.764254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:52.764226    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d374cdd2-4367-4745-8290-33a05145a48d-maas-api-xks-ca" pod "d374cdd2-4367-4745-8290-33a05145a48d" (UID: "d374cdd2-4367-4745-8290-33a05145a48d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:43:52.765504 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:52.765476    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d374cdd2-4367-4745-8290-33a05145a48d-kube-api-access-v67vc" pod "d374cdd2-4367-4745-8290-33a05145a48d" (UID: "d374cdd2-4367-4745-8290-33a05145a48d"). InnerVolumeSpecName "kube-api-access-v67vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:43:52.872540 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:52.871459    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v67vc\" (UniqueName: \"kubernetes.io/projected/d374cdd2-4367-4745-8290-33a05145a48d-kube-api-access-v67vc\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:43:52.872540 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:52.871496    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d374cdd2-4367-4745-8290-33a05145a48d-maas-api-xks-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:43:53.485559 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:53.485523    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2" event={"ID":"d374cdd2-4367-4745-8290-33a05145a48d","Type":"ContainerDied","Data":"02764320ea1a8c9b3eb65bd4a4259624fce75128885fc49e5b7f7a115731f653"}
Oct 31 13:43:53.485765 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:53.485575    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"
Oct 31 13:43:53.485765 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:43:53.485600    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02764320ea1a8c9b3eb65bd4a4259624fce75128885fc49e5b7f7a115731f653"
Oct 31 13:44:00.962049 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:00.962008    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"]
Oct 31 13:44:00.962603 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:00.962282    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66" podUID="040f7f90-43a7-4162-8de4-d17a85cb15ca" containerName="istio-proxy" containerID="cri-o://581236f086a6ab6a23cb40b8440f34fcf86efc2898354327b563c6f577646f28" gracePeriod=30
Oct 31 13:44:01.324067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:01.323970    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"]
Oct 31 13:44:01.324268 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:01.324215    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy" containerID="cri-o://79bed033ec609fdfd49624c999abf5b779e2be762f5a22595e5fb2ab17ece647" gracePeriod=30
Oct 31 13:44:03.937163 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:03.937114    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2" podUID="e003dfad-238c-47f3-aa19-696ca2c1264a" containerName="main" containerID="cri-o://eb36704f525a37940eda180b3d3a81318f6c221a5daf2c5510d3509290bd2284" gracePeriod=45
Oct 31 13:44:04.297698 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.297666    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:44:04.467177 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.467140    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-dshm\") pod \"e003dfad-238c-47f3-aa19-696ca2c1264a\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") "
Oct 31 13:44:04.467407 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.467254    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e003dfad-238c-47f3-aa19-696ca2c1264a-kube-api-access-fgs9k\" (UniqueName: \"kubernetes.io/projected/e003dfad-238c-47f3-aa19-696ca2c1264a-kube-api-access-fgs9k\") pod \"e003dfad-238c-47f3-aa19-696ca2c1264a\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") "
Oct 31 13:44:04.467407 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.467398    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-tmp-dir\") pod \"e003dfad-238c-47f3-aa19-696ca2c1264a\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") "
Oct 31 13:44:04.468533 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.467419    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-home\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-home\") pod \"e003dfad-238c-47f3-aa19-696ca2c1264a\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") "
Oct 31 13:44:04.468533 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.467542    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-model-cache\") pod \"e003dfad-238c-47f3-aa19-696ca2c1264a\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") "
Oct 31 13:44:04.468533 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.467568    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e003dfad-238c-47f3-aa19-696ca2c1264a-tls-certs\" (UniqueName: \"kubernetes.io/secret/e003dfad-238c-47f3-aa19-696ca2c1264a-tls-certs\") pod \"e003dfad-238c-47f3-aa19-696ca2c1264a\" (UID: \"e003dfad-238c-47f3-aa19-696ca2c1264a\") "
Oct 31 13:44:04.468533 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.467661    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-tmp-dir" pod "e003dfad-238c-47f3-aa19-696ca2c1264a" (UID: "e003dfad-238c-47f3-aa19-696ca2c1264a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:04.468533 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.467747    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-home" pod "e003dfad-238c-47f3-aa19-696ca2c1264a" (UID: "e003dfad-238c-47f3-aa19-696ca2c1264a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:04.468533 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.467765    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-model-cache" pod "e003dfad-238c-47f3-aa19-696ca2c1264a" (UID: "e003dfad-238c-47f3-aa19-696ca2c1264a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:04.468533 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.468429    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:04.468533 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.468452    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:04.468533 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.468465    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:04.469655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.469630    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-dshm" pod "e003dfad-238c-47f3-aa19-696ca2c1264a" (UID: "e003dfad-238c-47f3-aa19-696ca2c1264a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:04.469855 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.469729    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e003dfad-238c-47f3-aa19-696ca2c1264a-tls-certs" pod "e003dfad-238c-47f3-aa19-696ca2c1264a" (UID: "e003dfad-238c-47f3-aa19-696ca2c1264a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:44:04.469855 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.469816    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e003dfad-238c-47f3-aa19-696ca2c1264a-kube-api-access-fgs9k" pod "e003dfad-238c-47f3-aa19-696ca2c1264a" (UID: "e003dfad-238c-47f3-aa19-696ca2c1264a"). InnerVolumeSpecName "kube-api-access-fgs9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:04.570494 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.570467    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e003dfad-238c-47f3-aa19-696ca2c1264a-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:04.570494 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.570493    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgs9k\" (UniqueName: \"kubernetes.io/projected/e003dfad-238c-47f3-aa19-696ca2c1264a-kube-api-access-fgs9k\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:04.570690 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.570504    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e003dfad-238c-47f3-aa19-696ca2c1264a-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:04.670740 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.670699    4660 generic.go:356] "Generic (PLEG): container finished" podID="e003dfad-238c-47f3-aa19-696ca2c1264a" containerID="eb36704f525a37940eda180b3d3a81318f6c221a5daf2c5510d3509290bd2284" exitCode=0
Oct 31 13:44:04.670935 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.670784    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"
Oct 31 13:44:04.670935 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.670784    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2" event={"ID":"e003dfad-238c-47f3-aa19-696ca2c1264a","Type":"ContainerDied","Data":"eb36704f525a37940eda180b3d3a81318f6c221a5daf2c5510d3509290bd2284"}
Oct 31 13:44:04.670935 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.670837    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2" event={"ID":"e003dfad-238c-47f3-aa19-696ca2c1264a","Type":"ContainerDied","Data":"8e9dc81a25f759cdc5a7f1d0088c700f18e8daf2737ce50c4f44d50579e57164"}
Oct 31 13:44:04.670935 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.670857    4660 scope.go:122] "RemoveContainer" containerID="eb36704f525a37940eda180b3d3a81318f6c221a5daf2c5510d3509290bd2284"
Oct 31 13:44:04.683187 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.683169    4660 scope.go:122] "RemoveContainer" containerID="eb36704f525a37940eda180b3d3a81318f6c221a5daf2c5510d3509290bd2284"
Oct 31 13:44:04.683501 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:44:04.683467    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb36704f525a37940eda180b3d3a81318f6c221a5daf2c5510d3509290bd2284\": container with ID starting with eb36704f525a37940eda180b3d3a81318f6c221a5daf2c5510d3509290bd2284 not found: ID does not exist" containerID="eb36704f525a37940eda180b3d3a81318f6c221a5daf2c5510d3509290bd2284"
Oct 31 13:44:04.683501 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.683495    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb36704f525a37940eda180b3d3a81318f6c221a5daf2c5510d3509290bd2284"} err="failed to get container status \"eb36704f525a37940eda180b3d3a81318f6c221a5daf2c5510d3509290bd2284\": rpc error: code = NotFound desc = could not find container \"eb36704f525a37940eda180b3d3a81318f6c221a5daf2c5510d3509290bd2284\": container with ID starting with eb36704f525a37940eda180b3d3a81318f6c221a5daf2c5510d3509290bd2284 not found: ID does not exist"
Oct 31 13:44:04.718344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.717634    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"]
Oct 31 13:44:04.726458 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:04.726427    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2"]
Oct 31 13:44:05.176658 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:05.176618    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d4b52a/test-model-d4b52a-kserve-9cd9fb59-r2ck2" podUID="e003dfad-238c-47f3-aa19-696ca2c1264a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:44:06.339256 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.339208    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:44:06.515419 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.515003    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-data\") pod \"040f7f90-43a7-4162-8de4-d17a85cb15ca\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") "
Oct 31 13:44:06.515682 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.515594    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040f7f90-43a7-4162-8de4-d17a85cb15ca-kube-api-access-s45lx\" (UniqueName: \"kubernetes.io/projected/040f7f90-43a7-4162-8de4-d17a85cb15ca-kube-api-access-s45lx\") pod \"040f7f90-43a7-4162-8de4-d17a85cb15ca\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") "
Oct 31 13:44:06.515869 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.515809    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/040f7f90-43a7-4162-8de4-d17a85cb15ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040f7f90-43a7-4162-8de4-d17a85cb15ca-istiod-ca-cert\") pod \"040f7f90-43a7-4162-8de4-d17a85cb15ca\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") "
Oct 31 13:44:06.516019 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.515989    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-token\" (UniqueName: \"kubernetes.io/projected/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-token\") pod \"040f7f90-43a7-4162-8de4-d17a85cb15ca\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") "
Oct 31 13:44:06.516116 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.516075    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-credential-socket\") pod \"040f7f90-43a7-4162-8de4-d17a85cb15ca\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") "
Oct 31 13:44:06.516184 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.516121    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-envoy\") pod \"040f7f90-43a7-4162-8de4-d17a85cb15ca\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") "
Oct 31 13:44:06.516242 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.516189    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-workload-socket\") pod \"040f7f90-43a7-4162-8de4-d17a85cb15ca\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") "
Oct 31 13:44:06.516316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.516266    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-podinfo\") pod \"040f7f90-43a7-4162-8de4-d17a85cb15ca\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") "
Oct 31 13:44:06.516385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.516330    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-workload-certs\") pod \"040f7f90-43a7-4162-8de4-d17a85cb15ca\" (UID: \"040f7f90-43a7-4162-8de4-d17a85cb15ca\") "
Oct 31 13:44:06.519084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.516616    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-data" pod "040f7f90-43a7-4162-8de4-d17a85cb15ca" (UID: "040f7f90-43a7-4162-8de4-d17a85cb15ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:06.519084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.517825    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-workload-socket" pod "040f7f90-43a7-4162-8de4-d17a85cb15ca" (UID: "040f7f90-43a7-4162-8de4-d17a85cb15ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:06.519084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.518160    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040f7f90-43a7-4162-8de4-d17a85cb15ca-istiod-ca-cert" pod "040f7f90-43a7-4162-8de4-d17a85cb15ca" (UID: "040f7f90-43a7-4162-8de4-d17a85cb15ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:06.519084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.518742    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-credential-socket" pod "040f7f90-43a7-4162-8de4-d17a85cb15ca" (UID: "040f7f90-43a7-4162-8de4-d17a85cb15ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:06.519084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.519044    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-workload-certs" pod "040f7f90-43a7-4162-8de4-d17a85cb15ca" (UID: "040f7f90-43a7-4162-8de4-d17a85cb15ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:06.521188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.519435    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040f7f90-43a7-4162-8de4-d17a85cb15ca-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.521188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.519459    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.521188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.519474    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.521188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.519486    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.521188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.519500    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.531043 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.526017    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040f7f90-43a7-4162-8de4-d17a85cb15ca-kube-api-access-s45lx" pod "040f7f90-43a7-4162-8de4-d17a85cb15ca" (UID: "040f7f90-43a7-4162-8de4-d17a85cb15ca"). InnerVolumeSpecName "kube-api-access-s45lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:06.531043 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.526222    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-podinfo" pod "040f7f90-43a7-4162-8de4-d17a85cb15ca" (UID: "040f7f90-43a7-4162-8de4-d17a85cb15ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:44:06.531043 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.526222    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-envoy" pod "040f7f90-43a7-4162-8de4-d17a85cb15ca" (UID: "040f7f90-43a7-4162-8de4-d17a85cb15ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:06.531043 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.526537    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-token" pod "040f7f90-43a7-4162-8de4-d17a85cb15ca" (UID: "040f7f90-43a7-4162-8de4-d17a85cb15ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:06.540399 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.540361    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e003dfad-238c-47f3-aa19-696ca2c1264a" path="/var/lib/kubelet/pods/e003dfad-238c-47f3-aa19-696ca2c1264a/volumes"
Oct 31 13:44:06.622731 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.621960    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s45lx\" (UniqueName: \"kubernetes.io/projected/040f7f90-43a7-4162-8de4-d17a85cb15ca-kube-api-access-s45lx\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.622731 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.622080    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.622731 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.622099    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.622731 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.622113    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040f7f90-43a7-4162-8de4-d17a85cb15ca-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.659379 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.659349    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:44:06.697091 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.697055    4660 generic.go:356] "Generic (PLEG): container finished" podID="040f7f90-43a7-4162-8de4-d17a85cb15ca" containerID="581236f086a6ab6a23cb40b8440f34fcf86efc2898354327b563c6f577646f28" exitCode=0
Oct 31 13:44:06.697276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.697112    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"
Oct 31 13:44:06.697276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.697142    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66" event={"ID":"040f7f90-43a7-4162-8de4-d17a85cb15ca","Type":"ContainerDied","Data":"581236f086a6ab6a23cb40b8440f34fcf86efc2898354327b563c6f577646f28"}
Oct 31 13:44:06.697276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.697204    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66" event={"ID":"040f7f90-43a7-4162-8de4-d17a85cb15ca","Type":"ContainerDied","Data":"b8bcf7086aba2ab61da355341d418d0058a492bf2b5b79fcef6adb8536346243"}
Oct 31 13:44:06.697276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.697234    4660 scope.go:122] "RemoveContainer" containerID="581236f086a6ab6a23cb40b8440f34fcf86efc2898354327b563c6f577646f28"
Oct 31 13:44:06.699425 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.699400    4660 generic.go:356] "Generic (PLEG): container finished" podID="1daa31f8-4e70-413f-9529-0a67eef57732" containerID="79bed033ec609fdfd49624c999abf5b779e2be762f5a22595e5fb2ab17ece647" exitCode=0
Oct 31 13:44:06.699730 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.699705    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" event={"ID":"1daa31f8-4e70-413f-9529-0a67eef57732","Type":"ContainerDied","Data":"79bed033ec609fdfd49624c999abf5b779e2be762f5a22595e5fb2ab17ece647"}
Oct 31 13:44:06.699865 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.699781    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw" event={"ID":"1daa31f8-4e70-413f-9529-0a67eef57732","Type":"ContainerDied","Data":"a7913f5e399d41ee3b46b8ae7e7f8e4d010f7d1240db921d537adefe90fb5b07"}
Oct 31 13:44:06.699929 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.699909    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"
Oct 31 13:44:06.713667 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.713631    4660 scope.go:122] "RemoveContainer" containerID="581236f086a6ab6a23cb40b8440f34fcf86efc2898354327b563c6f577646f28"
Oct 31 13:44:06.714098 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:44:06.714069    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"581236f086a6ab6a23cb40b8440f34fcf86efc2898354327b563c6f577646f28\": container with ID starting with 581236f086a6ab6a23cb40b8440f34fcf86efc2898354327b563c6f577646f28 not found: ID does not exist" containerID="581236f086a6ab6a23cb40b8440f34fcf86efc2898354327b563c6f577646f28"
Oct 31 13:44:06.714206 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.714099    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"581236f086a6ab6a23cb40b8440f34fcf86efc2898354327b563c6f577646f28"} err="failed to get container status \"581236f086a6ab6a23cb40b8440f34fcf86efc2898354327b563c6f577646f28\": rpc error: code = NotFound desc = could not find container \"581236f086a6ab6a23cb40b8440f34fcf86efc2898354327b563c6f577646f28\": container with ID starting with 581236f086a6ab6a23cb40b8440f34fcf86efc2898354327b563c6f577646f28 not found: ID does not exist"
Oct 31 13:44:06.714206 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.714116    4660 scope.go:122] "RemoveContainer" containerID="79bed033ec609fdfd49624c999abf5b779e2be762f5a22595e5fb2ab17ece647"
Oct 31 13:44:06.722807 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.722784    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1daa31f8-4e70-413f-9529-0a67eef57732-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1daa31f8-4e70-413f-9529-0a67eef57732-istiod-ca-cert\") pod \"1daa31f8-4e70-413f-9529-0a67eef57732\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") "
Oct 31 13:44:06.723404 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.723385    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-istio-data\") pod \"1daa31f8-4e70-413f-9529-0a67eef57732\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") "
Oct 31 13:44:06.724006 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.723988    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-workload-socket\") pod \"1daa31f8-4e70-413f-9529-0a67eef57732\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") "
Oct 31 13:44:06.724418 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.724401    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1daa31f8-4e70-413f-9529-0a67eef57732-kube-api-access-tn4j2\" (UniqueName: \"kubernetes.io/projected/1daa31f8-4e70-413f-9529-0a67eef57732-kube-api-access-tn4j2\") pod \"1daa31f8-4e70-413f-9529-0a67eef57732\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") "
Oct 31 13:44:06.725017 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.723258    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1daa31f8-4e70-413f-9529-0a67eef57732-istiod-ca-cert" pod "1daa31f8-4e70-413f-9529-0a67eef57732" (UID: "1daa31f8-4e70-413f-9529-0a67eef57732"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:06.725160 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.723819    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-istio-data" pod "1daa31f8-4e70-413f-9529-0a67eef57732" (UID: "1daa31f8-4e70-413f-9529-0a67eef57732"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:06.725242 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.724346    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-workload-socket" pod "1daa31f8-4e70-413f-9529-0a67eef57732" (UID: "1daa31f8-4e70-413f-9529-0a67eef57732"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:06.725428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.725412    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1daa31f8-4e70-413f-9529-0a67eef57732-istio-token\" (UniqueName: \"kubernetes.io/projected/1daa31f8-4e70-413f-9529-0a67eef57732-istio-token\") pod \"1daa31f8-4e70-413f-9529-0a67eef57732\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") "
Oct 31 13:44:06.728195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.728174    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-credential-socket" pod "1daa31f8-4e70-413f-9529-0a67eef57732" (UID: "1daa31f8-4e70-413f-9529-0a67eef57732"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:06.728834 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.728794    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1daa31f8-4e70-413f-9529-0a67eef57732-kube-api-access-tn4j2" pod "1daa31f8-4e70-413f-9529-0a67eef57732" (UID: "1daa31f8-4e70-413f-9529-0a67eef57732"). InnerVolumeSpecName "kube-api-access-tn4j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:06.730021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.728369    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-credential-socket\") pod \"1daa31f8-4e70-413f-9529-0a67eef57732\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") "
Oct 31 13:44:06.730021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.729053    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-workload-certs\") pod \"1daa31f8-4e70-413f-9529-0a67eef57732\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") "
Oct 31 13:44:06.730021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.729353    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-workload-certs" pod "1daa31f8-4e70-413f-9529-0a67eef57732" (UID: "1daa31f8-4e70-413f-9529-0a67eef57732"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:06.730021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.729487    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1daa31f8-4e70-413f-9529-0a67eef57732-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1daa31f8-4e70-413f-9529-0a67eef57732-istio-podinfo\") pod \"1daa31f8-4e70-413f-9529-0a67eef57732\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") "
Oct 31 13:44:06.731893 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.731547    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-istio-envoy\") pod \"1daa31f8-4e70-413f-9529-0a67eef57732\" (UID: \"1daa31f8-4e70-413f-9529-0a67eef57732\") "
Oct 31 13:44:06.733480 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.733454    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-istio-envoy" pod "1daa31f8-4e70-413f-9529-0a67eef57732" (UID: "1daa31f8-4e70-413f-9529-0a67eef57732"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:06.733564 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.733476    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.733564 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.733497    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.733564 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.733511    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1daa31f8-4e70-413f-9529-0a67eef57732-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.733564 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.733524    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.733564 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.733537    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.733564 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.733551    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4j2\" (UniqueName: \"kubernetes.io/projected/1daa31f8-4e70-413f-9529-0a67eef57732-kube-api-access-tn4j2\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.734759 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.734722    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1daa31f8-4e70-413f-9529-0a67eef57732-istio-podinfo" pod "1daa31f8-4e70-413f-9529-0a67eef57732" (UID: "1daa31f8-4e70-413f-9529-0a67eef57732"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:44:06.735846 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.735821    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1daa31f8-4e70-413f-9529-0a67eef57732-istio-token" pod "1daa31f8-4e70-413f-9529-0a67eef57732" (UID: "1daa31f8-4e70-413f-9529-0a67eef57732"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:06.743903 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.743882    4660 scope.go:122] "RemoveContainer" containerID="79bed033ec609fdfd49624c999abf5b779e2be762f5a22595e5fb2ab17ece647"
Oct 31 13:44:06.744283 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:44:06.744257    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79bed033ec609fdfd49624c999abf5b779e2be762f5a22595e5fb2ab17ece647\": container with ID starting with 79bed033ec609fdfd49624c999abf5b779e2be762f5a22595e5fb2ab17ece647 not found: ID does not exist" containerID="79bed033ec609fdfd49624c999abf5b779e2be762f5a22595e5fb2ab17ece647"
Oct 31 13:44:06.744373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.744310    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79bed033ec609fdfd49624c999abf5b779e2be762f5a22595e5fb2ab17ece647"} err="failed to get container status \"79bed033ec609fdfd49624c999abf5b779e2be762f5a22595e5fb2ab17ece647\": rpc error: code = NotFound desc = could not find container \"79bed033ec609fdfd49624c999abf5b779e2be762f5a22595e5fb2ab17ece647\": container with ID starting with 79bed033ec609fdfd49624c999abf5b779e2be762f5a22595e5fb2ab17ece647 not found: ID does not exist"
Oct 31 13:44:06.747545 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.747520    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"]
Oct 31 13:44:06.752769 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.752741    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rkm66"]
Oct 31 13:44:06.836131 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.836038    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1daa31f8-4e70-413f-9529-0a67eef57732-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.836131 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.836077    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1daa31f8-4e70-413f-9529-0a67eef57732-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:06.836131 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:06.836092    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1daa31f8-4e70-413f-9529-0a67eef57732-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:07.071178 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:07.071137    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"]
Oct 31 13:44:07.081722 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:07.081676    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-r6sfw"]
Oct 31 13:44:08.531306 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:08.531262    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="040f7f90-43a7-4162-8de4-d17a85cb15ca" path="/var/lib/kubelet/pods/040f7f90-43a7-4162-8de4-d17a85cb15ca/volumes"
Oct 31 13:44:08.531887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:08.531870    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" path="/var/lib/kubelet/pods/1daa31f8-4e70-413f-9529-0a67eef57732/volumes"
Oct 31 13:44:11.840909 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:11.840851    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"]
Oct 31 13:44:11.841725 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:11.841672    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy" containerID="cri-o://cc7d377c8c85036a57ca2bfe3a90c15a085bf189021f88a04b322a892afce7ec" gracePeriod=30
Oct 31 13:44:15.836978 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:15.836937    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"]
Oct 31 13:44:15.837527 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:15.837178    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy" containerID="cri-o://125ded9fb17944d870b98a5607223b07032c8ac8bceb93b3bd11f11a48703e57" gracePeriod=30
Oct 31 13:44:15.997867 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:15.997827    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"]
Oct 31 13:44:15.998095 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:15.998070    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg" podUID="5c132a2a-1e7e-4858-80fb-376083a5b0f7" containerName="istio-proxy" containerID="cri-o://761ebd16829fb399522f86d68a5bbdff757a5961e5ae32f5ddcaaa6d8d1ce17b" gracePeriod=30
Oct 31 13:44:16.006726 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:16.006693    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"]
Oct 31 13:44:16.007408 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:16.006922    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy" containerID="cri-o://235be30bad4a3ebbd8ed56f635ac3a1b3ccaf8ff26f881a64437ee94c0c7323d" gracePeriod=30
Oct 31 13:44:16.223481 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:16.223443    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"]
Oct 31 13:44:16.223700 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:16.223647    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy" containerID="cri-o://1deba25e69ae1f08e3994501aa5bc4ba375416444e71749d9fba2f7e93733caf" gracePeriod=30
Oct 31 13:44:16.363211 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:16.363173    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:44:16.363493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:16.363230    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:44:16.371521 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:16.371486    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"]
Oct 31 13:44:16.371764 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:16.371735    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6" podUID="8906fc94-2d02-49d6-8ab0-c578e4b7391a" containerName="istio-proxy" containerID="cri-o://18098a7c79564dd9ba56e331626f241cf54a7087d09bf719b420dbd0f82a62e6" gracePeriod=30
Oct 31 13:44:17.163937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.163910    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:44:17.236166 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.236134    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-credential-socket\") pod \"2060533a-c13a-4a2e-bcd0-c904960ed76c\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") "
Oct 31 13:44:17.236349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.236187    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-workload-socket\") pod \"2060533a-c13a-4a2e-bcd0-c904960ed76c\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") "
Oct 31 13:44:17.236349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.236335    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-token\" (UniqueName: \"kubernetes.io/projected/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-token\") pod \"2060533a-c13a-4a2e-bcd0-c904960ed76c\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") "
Oct 31 13:44:17.236488 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.236387    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-envoy\") pod \"2060533a-c13a-4a2e-bcd0-c904960ed76c\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") "
Oct 31 13:44:17.236544 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.236501    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-workload-certs\") pod \"2060533a-c13a-4a2e-bcd0-c904960ed76c\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") "
Oct 31 13:44:17.236544 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.236514    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-credential-socket" pod "2060533a-c13a-4a2e-bcd0-c904960ed76c" (UID: "2060533a-c13a-4a2e-bcd0-c904960ed76c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:17.236648 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.236616    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2060533a-c13a-4a2e-bcd0-c904960ed76c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2060533a-c13a-4a2e-bcd0-c904960ed76c-istiod-ca-cert\") pod \"2060533a-c13a-4a2e-bcd0-c904960ed76c\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") "
Oct 31 13:44:17.236699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.236670    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-data\") pod \"2060533a-c13a-4a2e-bcd0-c904960ed76c\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") "
Oct 31 13:44:17.236776 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.236745    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-podinfo\") pod \"2060533a-c13a-4a2e-bcd0-c904960ed76c\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") "
Oct 31 13:44:17.237139 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.237113    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-workload-certs" pod "2060533a-c13a-4a2e-bcd0-c904960ed76c" (UID: "2060533a-c13a-4a2e-bcd0-c904960ed76c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:17.237281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.237260    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2060533a-c13a-4a2e-bcd0-c904960ed76c-kube-api-access-zvb66\" (UniqueName: \"kubernetes.io/projected/2060533a-c13a-4a2e-bcd0-c904960ed76c-kube-api-access-zvb66\") pod \"2060533a-c13a-4a2e-bcd0-c904960ed76c\" (UID: \"2060533a-c13a-4a2e-bcd0-c904960ed76c\") "
Oct 31 13:44:17.238733 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.238704    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:17.238855 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.238736    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:17.238855 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.237227    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2060533a-c13a-4a2e-bcd0-c904960ed76c-istiod-ca-cert" pod "2060533a-c13a-4a2e-bcd0-c904960ed76c" (UID: "2060533a-c13a-4a2e-bcd0-c904960ed76c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:17.240240 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.240020    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-token" pod "2060533a-c13a-4a2e-bcd0-c904960ed76c" (UID: "2060533a-c13a-4a2e-bcd0-c904960ed76c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:17.240577 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.240407    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-workload-socket" pod "2060533a-c13a-4a2e-bcd0-c904960ed76c" (UID: "2060533a-c13a-4a2e-bcd0-c904960ed76c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:17.240916 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.240849    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-data" pod "2060533a-c13a-4a2e-bcd0-c904960ed76c" (UID: "2060533a-c13a-4a2e-bcd0-c904960ed76c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:17.241522 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.241488    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-envoy" pod "2060533a-c13a-4a2e-bcd0-c904960ed76c" (UID: "2060533a-c13a-4a2e-bcd0-c904960ed76c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:17.242458 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.242422    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2060533a-c13a-4a2e-bcd0-c904960ed76c-kube-api-access-zvb66" pod "2060533a-c13a-4a2e-bcd0-c904960ed76c" (UID: "2060533a-c13a-4a2e-bcd0-c904960ed76c"). InnerVolumeSpecName "kube-api-access-zvb66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:17.247728 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.247699    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-podinfo" pod "2060533a-c13a-4a2e-bcd0-c904960ed76c" (UID: "2060533a-c13a-4a2e-bcd0-c904960ed76c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:44:17.343354 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.343319    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvb66\" (UniqueName: \"kubernetes.io/projected/2060533a-c13a-4a2e-bcd0-c904960ed76c-kube-api-access-zvb66\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:17.343354 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.343352    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:17.344168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.343372    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:17.344168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.343384    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:17.344168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.343397    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2060533a-c13a-4a2e-bcd0-c904960ed76c-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:17.344168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.343416    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:17.344168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.343429    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2060533a-c13a-4a2e-bcd0-c904960ed76c-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:17.902331 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.902277    4660 generic.go:356] "Generic (PLEG): container finished" podID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerID="cc7d377c8c85036a57ca2bfe3a90c15a085bf189021f88a04b322a892afce7ec" exitCode=0
Oct 31 13:44:17.902541 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.902361    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" event={"ID":"2060533a-c13a-4a2e-bcd0-c904960ed76c","Type":"ContainerDied","Data":"cc7d377c8c85036a57ca2bfe3a90c15a085bf189021f88a04b322a892afce7ec"}
Oct 31 13:44:17.902541 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.902398    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"
Oct 31 13:44:17.902541 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.902428    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q" event={"ID":"2060533a-c13a-4a2e-bcd0-c904960ed76c","Type":"ContainerDied","Data":"77b62480f6bac77fc765f3e1bed3a4f35c7ef0c9a5c431ca5635d64734eed5e0"}
Oct 31 13:44:17.902541 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.902456    4660 scope.go:122] "RemoveContainer" containerID="cc7d377c8c85036a57ca2bfe3a90c15a085bf189021f88a04b322a892afce7ec"
Oct 31 13:44:17.920864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.920841    4660 scope.go:122] "RemoveContainer" containerID="cc7d377c8c85036a57ca2bfe3a90c15a085bf189021f88a04b322a892afce7ec"
Oct 31 13:44:17.921272 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:44:17.921244    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7d377c8c85036a57ca2bfe3a90c15a085bf189021f88a04b322a892afce7ec\": container with ID starting with cc7d377c8c85036a57ca2bfe3a90c15a085bf189021f88a04b322a892afce7ec not found: ID does not exist" containerID="cc7d377c8c85036a57ca2bfe3a90c15a085bf189021f88a04b322a892afce7ec"
Oct 31 13:44:17.921435 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.921286    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7d377c8c85036a57ca2bfe3a90c15a085bf189021f88a04b322a892afce7ec"} err="failed to get container status \"cc7d377c8c85036a57ca2bfe3a90c15a085bf189021f88a04b322a892afce7ec\": rpc error: code = NotFound desc = could not find container \"cc7d377c8c85036a57ca2bfe3a90c15a085bf189021f88a04b322a892afce7ec\": container with ID starting with cc7d377c8c85036a57ca2bfe3a90c15a085bf189021f88a04b322a892afce7ec not found: ID does not exist"
Oct 31 13:44:17.937458 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.937433    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"]
Oct 31 13:44:17.958359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:17.950742    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5c26q"]
Oct 31 13:44:18.532837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:18.532803    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" path="/var/lib/kubelet/pods/2060533a-c13a-4a2e-bcd0-c904960ed76c/volumes"
Oct 31 13:44:20.741097 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:44:20.741061    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:44:20.972857 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:20.972822    4660 generic.go:356] "Generic (PLEG): container finished" podID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerID="125ded9fb17944d870b98a5607223b07032c8ac8bceb93b3bd11f11a48703e57" exitCode=0
Oct 31 13:44:20.973118 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:20.973099    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" event={"ID":"16adae6b-a455-42f5-a51b-f9c018eec1cc","Type":"ContainerDied","Data":"125ded9fb17944d870b98a5607223b07032c8ac8bceb93b3bd11f11a48703e57"}
Oct 31 13:44:21.217158 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.212362    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:44:21.291358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.291225    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-credential-socket\") pod \"16adae6b-a455-42f5-a51b-f9c018eec1cc\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") "
Oct 31 13:44:21.291358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.291319    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-podinfo\") pod \"16adae6b-a455-42f5-a51b-f9c018eec1cc\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") "
Oct 31 13:44:21.291582 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.291365    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16adae6b-a455-42f5-a51b-f9c018eec1cc-kube-api-access-xqnxz\" (UniqueName: \"kubernetes.io/projected/16adae6b-a455-42f5-a51b-f9c018eec1cc-kube-api-access-xqnxz\") pod \"16adae6b-a455-42f5-a51b-f9c018eec1cc\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") "
Oct 31 13:44:21.291582 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.291442    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-data\") pod \"16adae6b-a455-42f5-a51b-f9c018eec1cc\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") "
Oct 31 13:44:21.291582 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.291493    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16adae6b-a455-42f5-a51b-f9c018eec1cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16adae6b-a455-42f5-a51b-f9c018eec1cc-istiod-ca-cert\") pod \"16adae6b-a455-42f5-a51b-f9c018eec1cc\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") "
Oct 31 13:44:21.291760 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.291634    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-envoy\") pod \"16adae6b-a455-42f5-a51b-f9c018eec1cc\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") "
Oct 31 13:44:21.291760 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.291714    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-workload-socket\") pod \"16adae6b-a455-42f5-a51b-f9c018eec1cc\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") "
Oct 31 13:44:21.291867 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.291758    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-workload-certs\") pod \"16adae6b-a455-42f5-a51b-f9c018eec1cc\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") "
Oct 31 13:44:21.291867 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.291832    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-token\" (UniqueName: \"kubernetes.io/projected/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-token\") pod \"16adae6b-a455-42f5-a51b-f9c018eec1cc\" (UID: \"16adae6b-a455-42f5-a51b-f9c018eec1cc\") "
Oct 31 13:44:21.293643 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.293603    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-data" pod "16adae6b-a455-42f5-a51b-f9c018eec1cc" (UID: "16adae6b-a455-42f5-a51b-f9c018eec1cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.294072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.294049    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-workload-socket" pod "16adae6b-a455-42f5-a51b-f9c018eec1cc" (UID: "16adae6b-a455-42f5-a51b-f9c018eec1cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.294505 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.294470    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-credential-socket" pod "16adae6b-a455-42f5-a51b-f9c018eec1cc" (UID: "16adae6b-a455-42f5-a51b-f9c018eec1cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.294642 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.294562    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16adae6b-a455-42f5-a51b-f9c018eec1cc-istiod-ca-cert" pod "16adae6b-a455-42f5-a51b-f9c018eec1cc" (UID: "16adae6b-a455-42f5-a51b-f9c018eec1cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:21.294755 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.294610    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-workload-certs" pod "16adae6b-a455-42f5-a51b-f9c018eec1cc" (UID: "16adae6b-a455-42f5-a51b-f9c018eec1cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.298365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.298096    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16adae6b-a455-42f5-a51b-f9c018eec1cc-kube-api-access-xqnxz" pod "16adae6b-a455-42f5-a51b-f9c018eec1cc" (UID: "16adae6b-a455-42f5-a51b-f9c018eec1cc"). InnerVolumeSpecName "kube-api-access-xqnxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:21.302742 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.302711    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-envoy" pod "16adae6b-a455-42f5-a51b-f9c018eec1cc" (UID: "16adae6b-a455-42f5-a51b-f9c018eec1cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.302823 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.302781    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-podinfo" pod "16adae6b-a455-42f5-a51b-f9c018eec1cc" (UID: "16adae6b-a455-42f5-a51b-f9c018eec1cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:44:21.304474 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.304450    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-token" pod "16adae6b-a455-42f5-a51b-f9c018eec1cc" (UID: "16adae6b-a455-42f5-a51b-f9c018eec1cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:21.385370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.381224    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:44:21.395146 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.394791    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.395804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.394826    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.395804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.395412    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.395804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.395435    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.395804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.395450    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.395804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.395463    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.395804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.395475    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqnxz\" (UniqueName: \"kubernetes.io/projected/16adae6b-a455-42f5-a51b-f9c018eec1cc-kube-api-access-xqnxz\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.395804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.395490    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16adae6b-a455-42f5-a51b-f9c018eec1cc-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.395804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.395504    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16adae6b-a455-42f5-a51b-f9c018eec1cc-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.500875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.497535    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:44:21.500875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.497907    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dd64e87-c152-497f-9460-075f624e4157-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd64e87-c152-497f-9460-075f624e4157-istio-podinfo\") pod \"8dd64e87-c152-497f-9460-075f624e4157\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") "
Oct 31 13:44:21.500875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.498050    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd64e87-c152-497f-9460-075f624e4157-kube-api-access-hzqgd\" (UniqueName: \"kubernetes.io/projected/8dd64e87-c152-497f-9460-075f624e4157-kube-api-access-hzqgd\") pod \"8dd64e87-c152-497f-9460-075f624e4157\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") "
Oct 31 13:44:21.500875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.498131    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-credential-socket\") pod \"8dd64e87-c152-497f-9460-075f624e4157\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") "
Oct 31 13:44:21.500875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.498203    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd64e87-c152-497f-9460-075f624e4157-istio-token\" (UniqueName: \"kubernetes.io/projected/8dd64e87-c152-497f-9460-075f624e4157-istio-token\") pod \"8dd64e87-c152-497f-9460-075f624e4157\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") "
Oct 31 13:44:21.500875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.498285    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-workload-certs\") pod \"8dd64e87-c152-497f-9460-075f624e4157\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") "
Oct 31 13:44:21.500875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.498350    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd64e87-c152-497f-9460-075f624e4157-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd64e87-c152-497f-9460-075f624e4157-istiod-ca-cert\") pod \"8dd64e87-c152-497f-9460-075f624e4157\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") "
Oct 31 13:44:21.500875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.498458    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-istio-data\") pod \"8dd64e87-c152-497f-9460-075f624e4157\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") "
Oct 31 13:44:21.500875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.498629    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-credential-socket" pod "8dd64e87-c152-497f-9460-075f624e4157" (UID: "8dd64e87-c152-497f-9460-075f624e4157"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.500875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.498638    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-workload-socket\") pod \"8dd64e87-c152-497f-9460-075f624e4157\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") "
Oct 31 13:44:21.500875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.498742    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-istio-envoy\") pod \"8dd64e87-c152-497f-9460-075f624e4157\" (UID: \"8dd64e87-c152-497f-9460-075f624e4157\") "
Oct 31 13:44:21.500875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.498849    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-workload-socket" pod "8dd64e87-c152-497f-9460-075f624e4157" (UID: "8dd64e87-c152-497f-9460-075f624e4157"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.500875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.499743    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.500875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.499770    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.504084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.502818    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-workload-certs" pod "8dd64e87-c152-497f-9460-075f624e4157" (UID: "8dd64e87-c152-497f-9460-075f624e4157"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.504084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.503897    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-istio-data" pod "8dd64e87-c152-497f-9460-075f624e4157" (UID: "8dd64e87-c152-497f-9460-075f624e4157"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.504084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.504056    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd64e87-c152-497f-9460-075f624e4157-istiod-ca-cert" pod "8dd64e87-c152-497f-9460-075f624e4157" (UID: "8dd64e87-c152-497f-9460-075f624e4157"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:21.504822 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.504783    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dd64e87-c152-497f-9460-075f624e4157-istio-podinfo" pod "8dd64e87-c152-497f-9460-075f624e4157" (UID: "8dd64e87-c152-497f-9460-075f624e4157"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:44:21.508308 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.507438    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd64e87-c152-497f-9460-075f624e4157-kube-api-access-hzqgd" pod "8dd64e87-c152-497f-9460-075f624e4157" (UID: "8dd64e87-c152-497f-9460-075f624e4157"). InnerVolumeSpecName "kube-api-access-hzqgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:21.581272 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.580387    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd64e87-c152-497f-9460-075f624e4157-istio-token" pod "8dd64e87-c152-497f-9460-075f624e4157" (UID: "8dd64e87-c152-497f-9460-075f624e4157"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:21.581272 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.581136    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-istio-envoy" pod "8dd64e87-c152-497f-9460-075f624e4157" (UID: "8dd64e87-c152-497f-9460-075f624e4157"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.603247    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-podinfo\") pod \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") "
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.603340    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-workload-socket\") pod \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") "
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.603380    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istiod-ca-cert\") pod \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") "
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.603448    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-token\" (UniqueName: \"kubernetes.io/projected/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-token\") pod \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") "
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.603485    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-credential-socket\") pod \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") "
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.603694    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c132a2a-1e7e-4858-80fb-376083a5b0f7-kube-api-access-r5dqr\" (UniqueName: \"kubernetes.io/projected/5c132a2a-1e7e-4858-80fb-376083a5b0f7-kube-api-access-r5dqr\") pod \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") "
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.603740    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-workload-socket" pod "5c132a2a-1e7e-4858-80fb-376083a5b0f7" (UID: "5c132a2a-1e7e-4858-80fb-376083a5b0f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.603823    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-envoy\") pod \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") "
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.603959    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-workload-certs\") pod \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") "
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.603991    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-data\") pod \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\" (UID: \"5c132a2a-1e7e-4858-80fb-376083a5b0f7\") "
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.605224    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd64e87-c152-497f-9460-075f624e4157-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.605249    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.605273    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd64e87-c152-497f-9460-075f624e4157-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.605306    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.605323    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd64e87-c152-497f-9460-075f624e4157-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.605337    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd64e87-c152-497f-9460-075f624e4157-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.605355    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.605371    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzqgd\" (UniqueName: \"kubernetes.io/projected/8dd64e87-c152-497f-9460-075f624e4157-kube-api-access-hzqgd\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.606435    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-podinfo" pod "5c132a2a-1e7e-4858-80fb-376083a5b0f7" (UID: "5c132a2a-1e7e-4858-80fb-376083a5b0f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.606663    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-credential-socket" pod "5c132a2a-1e7e-4858-80fb-376083a5b0f7" (UID: "5c132a2a-1e7e-4858-80fb-376083a5b0f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.608046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.606967    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istiod-ca-cert" pod "5c132a2a-1e7e-4858-80fb-376083a5b0f7" (UID: "5c132a2a-1e7e-4858-80fb-376083a5b0f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:21.609385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.609265    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-token" pod "5c132a2a-1e7e-4858-80fb-376083a5b0f7" (UID: "5c132a2a-1e7e-4858-80fb-376083a5b0f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:21.610142 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.609996    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-workload-certs" pod "5c132a2a-1e7e-4858-80fb-376083a5b0f7" (UID: "5c132a2a-1e7e-4858-80fb-376083a5b0f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.613347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.611385    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-data" pod "5c132a2a-1e7e-4858-80fb-376083a5b0f7" (UID: "5c132a2a-1e7e-4858-80fb-376083a5b0f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.625372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.622423    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c132a2a-1e7e-4858-80fb-376083a5b0f7-kube-api-access-r5dqr" pod "5c132a2a-1e7e-4858-80fb-376083a5b0f7" (UID: "5c132a2a-1e7e-4858-80fb-376083a5b0f7"). InnerVolumeSpecName "kube-api-access-r5dqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:21.625372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.622817    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-envoy" pod "5c132a2a-1e7e-4858-80fb-376083a5b0f7" (UID: "5c132a2a-1e7e-4858-80fb-376083a5b0f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.630168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.630145    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:44:21.706053 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.706020    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-token\" (UniqueName: \"kubernetes.io/projected/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-token\") pod \"5833e6de-e1f1-425c-865f-9293c4396f1e\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") "
Oct 31 13:44:21.706245 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.706158    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5833e6de-e1f1-425c-865f-9293c4396f1e-kube-api-access-hjxcr\" (UniqueName: \"kubernetes.io/projected/5833e6de-e1f1-425c-865f-9293c4396f1e-kube-api-access-hjxcr\") pod \"5833e6de-e1f1-425c-865f-9293c4396f1e\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") "
Oct 31 13:44:21.706245 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.706198    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-workload-certs\") pod \"5833e6de-e1f1-425c-865f-9293c4396f1e\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") "
Oct 31 13:44:21.706375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.706246    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-data\") pod \"5833e6de-e1f1-425c-865f-9293c4396f1e\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") "
Oct 31 13:44:21.706375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.706339    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-envoy\") pod \"5833e6de-e1f1-425c-865f-9293c4396f1e\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") "
Oct 31 13:44:21.706375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.706363    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-podinfo\") pod \"5833e6de-e1f1-425c-865f-9293c4396f1e\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") "
Oct 31 13:44:21.706521 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.706401    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-workload-socket\") pod \"5833e6de-e1f1-425c-865f-9293c4396f1e\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") "
Oct 31 13:44:21.706521 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.706456    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5833e6de-e1f1-425c-865f-9293c4396f1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5833e6de-e1f1-425c-865f-9293c4396f1e-istiod-ca-cert\") pod \"5833e6de-e1f1-425c-865f-9293c4396f1e\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") "
Oct 31 13:44:21.706625 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.706575    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-credential-socket\") pod \"5833e6de-e1f1-425c-865f-9293c4396f1e\" (UID: \"5833e6de-e1f1-425c-865f-9293c4396f1e\") "
Oct 31 13:44:21.706983 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.706957    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-data" pod "5833e6de-e1f1-425c-865f-9293c4396f1e" (UID: "5833e6de-e1f1-425c-865f-9293c4396f1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.707329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.707271    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-credential-socket" pod "5833e6de-e1f1-425c-865f-9293c4396f1e" (UID: "5833e6de-e1f1-425c-865f-9293c4396f1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.707473 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.707449    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.707552 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.707479    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.707552 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.707493    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.707552 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.707505    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.707552 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.707518    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.707552 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.707532    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5dqr\" (UniqueName: \"kubernetes.io/projected/5c132a2a-1e7e-4858-80fb-376083a5b0f7-kube-api-access-r5dqr\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.707552 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.707546    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.707872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.707560    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.707872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.707572    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.707872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.707585    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c132a2a-1e7e-4858-80fb-376083a5b0f7-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.707872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.707629    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-workload-socket" pod "5833e6de-e1f1-425c-865f-9293c4396f1e" (UID: "5833e6de-e1f1-425c-865f-9293c4396f1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.707872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.707647    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5833e6de-e1f1-425c-865f-9293c4396f1e-istiod-ca-cert" pod "5833e6de-e1f1-425c-865f-9293c4396f1e" (UID: "5833e6de-e1f1-425c-865f-9293c4396f1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:21.708526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.708499    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-workload-certs" pod "5833e6de-e1f1-425c-865f-9293c4396f1e" (UID: "5833e6de-e1f1-425c-865f-9293c4396f1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.711044 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.711021    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-podinfo" pod "5833e6de-e1f1-425c-865f-9293c4396f1e" (UID: "5833e6de-e1f1-425c-865f-9293c4396f1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:44:21.711858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.711838    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-envoy" pod "5833e6de-e1f1-425c-865f-9293c4396f1e" (UID: "5833e6de-e1f1-425c-865f-9293c4396f1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.711957 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.711908    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-token" pod "5833e6de-e1f1-425c-865f-9293c4396f1e" (UID: "5833e6de-e1f1-425c-865f-9293c4396f1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:21.717195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.717169    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5833e6de-e1f1-425c-865f-9293c4396f1e-kube-api-access-hjxcr" pod "5833e6de-e1f1-425c-865f-9293c4396f1e" (UID: "5833e6de-e1f1-425c-865f-9293c4396f1e"). InnerVolumeSpecName "kube-api-access-hjxcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:21.763514 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.763489    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:44:21.808355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.808321    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-podinfo\") pod \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") "
Oct 31 13:44:21.808558 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.808471    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-workload-certs\") pod \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") "
Oct 31 13:44:21.808558 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.808515    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-credential-socket\") pod \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") "
Oct 31 13:44:21.808693 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.808585    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istiod-ca-cert\") pod \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") "
Oct 31 13:44:21.808693 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.808629    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-workload-socket\") pod \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") "
Oct 31 13:44:21.808693 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.808689    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-data\") pod \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") "
Oct 31 13:44:21.808847 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.808754    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-envoy\") pod \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") "
Oct 31 13:44:21.808920 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.808898    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-token\" (UniqueName: \"kubernetes.io/projected/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-token\") pod \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") "
Oct 31 13:44:21.808980 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.808951    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8906fc94-2d02-49d6-8ab0-c578e4b7391a-kube-api-access-2x2np\" (UniqueName: \"kubernetes.io/projected/8906fc94-2d02-49d6-8ab0-c578e4b7391a-kube-api-access-2x2np\") pod \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\" (UID: \"8906fc94-2d02-49d6-8ab0-c578e4b7391a\") "
Oct 31 13:44:21.809187 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.809163    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istiod-ca-cert" pod "8906fc94-2d02-49d6-8ab0-c578e4b7391a" (UID: "8906fc94-2d02-49d6-8ab0-c578e4b7391a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:21.809514 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.809492    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-workload-certs" pod "8906fc94-2d02-49d6-8ab0-c578e4b7391a" (UID: "8906fc94-2d02-49d6-8ab0-c578e4b7391a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.809835 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.809788    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.809835 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.809814    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.809835 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.809828    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.810051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.809843    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5833e6de-e1f1-425c-865f-9293c4396f1e-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.810051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.809856    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5833e6de-e1f1-425c-865f-9293c4396f1e-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.810051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.809868    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.810051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.809882    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjxcr\" (UniqueName: \"kubernetes.io/projected/5833e6de-e1f1-425c-865f-9293c4396f1e-kube-api-access-hjxcr\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.810051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.809897    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.810051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.809910    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5833e6de-e1f1-425c-865f-9293c4396f1e-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.810578 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.810140    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-credential-socket" pod "8906fc94-2d02-49d6-8ab0-c578e4b7391a" (UID: "8906fc94-2d02-49d6-8ab0-c578e4b7391a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.810578 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.810352    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-data" pod "8906fc94-2d02-49d6-8ab0-c578e4b7391a" (UID: "8906fc94-2d02-49d6-8ab0-c578e4b7391a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.810578 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.810547    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-workload-socket" pod "8906fc94-2d02-49d6-8ab0-c578e4b7391a" (UID: "8906fc94-2d02-49d6-8ab0-c578e4b7391a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.812557 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.812237    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8906fc94-2d02-49d6-8ab0-c578e4b7391a-kube-api-access-2x2np" pod "8906fc94-2d02-49d6-8ab0-c578e4b7391a" (UID: "8906fc94-2d02-49d6-8ab0-c578e4b7391a"). InnerVolumeSpecName "kube-api-access-2x2np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:21.813079 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.813055    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-podinfo" pod "8906fc94-2d02-49d6-8ab0-c578e4b7391a" (UID: "8906fc94-2d02-49d6-8ab0-c578e4b7391a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:44:21.813079 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.813063    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-envoy" pod "8906fc94-2d02-49d6-8ab0-c578e4b7391a" (UID: "8906fc94-2d02-49d6-8ab0-c578e4b7391a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:21.814695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.814661    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-token" pod "8906fc94-2d02-49d6-8ab0-c578e4b7391a" (UID: "8906fc94-2d02-49d6-8ab0-c578e4b7391a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:21.911842 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.911804    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.911955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.911844    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.911955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.911860    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.911955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.911874    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x2np\" (UniqueName: \"kubernetes.io/projected/8906fc94-2d02-49d6-8ab0-c578e4b7391a-kube-api-access-2x2np\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.911955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.911889    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8906fc94-2d02-49d6-8ab0-c578e4b7391a-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.911955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.911905    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:21.911955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:21.911920    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8906fc94-2d02-49d6-8ab0-c578e4b7391a-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:22.028714 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.028676    4660 generic.go:356] "Generic (PLEG): container finished" podID="8dd64e87-c152-497f-9460-075f624e4157" containerID="235be30bad4a3ebbd8ed56f635ac3a1b3ccaf8ff26f881a64437ee94c0c7323d" exitCode=0
Oct 31 13:44:22.028986 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.028755    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"
Oct 31 13:44:22.029372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.028773    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" event={"ID":"8dd64e87-c152-497f-9460-075f624e4157","Type":"ContainerDied","Data":"235be30bad4a3ebbd8ed56f635ac3a1b3ccaf8ff26f881a64437ee94c0c7323d"}
Oct 31 13:44:22.029372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.029234    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8" event={"ID":"8dd64e87-c152-497f-9460-075f624e4157","Type":"ContainerDied","Data":"1477405b422e109c078580db11193539635ebd98bbd6e2133c1694ffb3f692eb"}
Oct 31 13:44:22.029372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.029270    4660 scope.go:122] "RemoveContainer" containerID="235be30bad4a3ebbd8ed56f635ac3a1b3ccaf8ff26f881a64437ee94c0c7323d"
Oct 31 13:44:22.037608 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.037575    4660 generic.go:356] "Generic (PLEG): container finished" podID="8906fc94-2d02-49d6-8ab0-c578e4b7391a" containerID="18098a7c79564dd9ba56e331626f241cf54a7087d09bf719b420dbd0f82a62e6" exitCode=0
Oct 31 13:44:22.037875 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.037857    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"
Oct 31 13:44:22.038740 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.038339    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6" event={"ID":"8906fc94-2d02-49d6-8ab0-c578e4b7391a","Type":"ContainerDied","Data":"18098a7c79564dd9ba56e331626f241cf54a7087d09bf719b420dbd0f82a62e6"}
Oct 31 13:44:22.038740 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.038394    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6" event={"ID":"8906fc94-2d02-49d6-8ab0-c578e4b7391a","Type":"ContainerDied","Data":"84cfca8a823127a666b1ad27986de9861787ca43379f6a107ec5b28ba89946a5"}
Oct 31 13:44:22.042528 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.042289    4660 generic.go:356] "Generic (PLEG): container finished" podID="5c132a2a-1e7e-4858-80fb-376083a5b0f7" containerID="761ebd16829fb399522f86d68a5bbdff757a5961e5ae32f5ddcaaa6d8d1ce17b" exitCode=0
Oct 31 13:44:22.042528 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.042453    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg" event={"ID":"5c132a2a-1e7e-4858-80fb-376083a5b0f7","Type":"ContainerDied","Data":"761ebd16829fb399522f86d68a5bbdff757a5961e5ae32f5ddcaaa6d8d1ce17b"}
Oct 31 13:44:22.042528 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.042496    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg" event={"ID":"5c132a2a-1e7e-4858-80fb-376083a5b0f7","Type":"ContainerDied","Data":"c6c865e14ca66768a5eeccee7a0fdb0306edb027fef6f2b9cc0ec0b090f610f6"}
Oct 31 13:44:22.042738 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.042565    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"
Oct 31 13:44:22.045807 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.045380    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9" event={"ID":"16adae6b-a455-42f5-a51b-f9c018eec1cc","Type":"ContainerDied","Data":"15e316b00b2b1a965c2b3f192e2065d5ba77cbe335c9f42d51b8e195237e5e78"}
Oct 31 13:44:22.045807 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.045407    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"
Oct 31 13:44:22.055003 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.054404    4660 generic.go:356] "Generic (PLEG): container finished" podID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerID="1deba25e69ae1f08e3994501aa5bc4ba375416444e71749d9fba2f7e93733caf" exitCode=0
Oct 31 13:44:22.055003 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.054504    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" event={"ID":"5833e6de-e1f1-425c-865f-9293c4396f1e","Type":"ContainerDied","Data":"1deba25e69ae1f08e3994501aa5bc4ba375416444e71749d9fba2f7e93733caf"}
Oct 31 13:44:22.055003 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.054540    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt" event={"ID":"5833e6de-e1f1-425c-865f-9293c4396f1e","Type":"ContainerDied","Data":"f2be43283708bc3ef2174d333a22562a0eb9b68d1fd8b29b79fece688106f6df"}
Oct 31 13:44:22.055003 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.054610    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"
Oct 31 13:44:22.079658 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.079545    4660 scope.go:122] "RemoveContainer" containerID="235be30bad4a3ebbd8ed56f635ac3a1b3ccaf8ff26f881a64437ee94c0c7323d"
Oct 31 13:44:22.080087 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:44:22.079980    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235be30bad4a3ebbd8ed56f635ac3a1b3ccaf8ff26f881a64437ee94c0c7323d\": container with ID starting with 235be30bad4a3ebbd8ed56f635ac3a1b3ccaf8ff26f881a64437ee94c0c7323d not found: ID does not exist" containerID="235be30bad4a3ebbd8ed56f635ac3a1b3ccaf8ff26f881a64437ee94c0c7323d"
Oct 31 13:44:22.080087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.080007    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235be30bad4a3ebbd8ed56f635ac3a1b3ccaf8ff26f881a64437ee94c0c7323d"} err="failed to get container status \"235be30bad4a3ebbd8ed56f635ac3a1b3ccaf8ff26f881a64437ee94c0c7323d\": rpc error: code = NotFound desc = could not find container \"235be30bad4a3ebbd8ed56f635ac3a1b3ccaf8ff26f881a64437ee94c0c7323d\": container with ID starting with 235be30bad4a3ebbd8ed56f635ac3a1b3ccaf8ff26f881a64437ee94c0c7323d not found: ID does not exist"
Oct 31 13:44:22.080087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.080023    4660 scope.go:122] "RemoveContainer" containerID="18098a7c79564dd9ba56e331626f241cf54a7087d09bf719b420dbd0f82a62e6"
Oct 31 13:44:22.092747 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.092718    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"]
Oct 31 13:44:22.105385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.103504    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-mqrn8"]
Oct 31 13:44:22.111261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.111241    4660 scope.go:122] "RemoveContainer" containerID="18098a7c79564dd9ba56e331626f241cf54a7087d09bf719b420dbd0f82a62e6"
Oct 31 13:44:22.111732 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:44:22.111706    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18098a7c79564dd9ba56e331626f241cf54a7087d09bf719b420dbd0f82a62e6\": container with ID starting with 18098a7c79564dd9ba56e331626f241cf54a7087d09bf719b420dbd0f82a62e6 not found: ID does not exist" containerID="18098a7c79564dd9ba56e331626f241cf54a7087d09bf719b420dbd0f82a62e6"
Oct 31 13:44:22.111826 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.111744    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18098a7c79564dd9ba56e331626f241cf54a7087d09bf719b420dbd0f82a62e6"} err="failed to get container status \"18098a7c79564dd9ba56e331626f241cf54a7087d09bf719b420dbd0f82a62e6\": rpc error: code = NotFound desc = could not find container \"18098a7c79564dd9ba56e331626f241cf54a7087d09bf719b420dbd0f82a62e6\": container with ID starting with 18098a7c79564dd9ba56e331626f241cf54a7087d09bf719b420dbd0f82a62e6 not found: ID does not exist"
Oct 31 13:44:22.111826 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.111771    4660 scope.go:122] "RemoveContainer" containerID="761ebd16829fb399522f86d68a5bbdff757a5961e5ae32f5ddcaaa6d8d1ce17b"
Oct 31 13:44:22.120317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.119370    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"]
Oct 31 13:44:22.126924 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.126826    4660 scope.go:122] "RemoveContainer" containerID="761ebd16829fb399522f86d68a5bbdff757a5961e5ae32f5ddcaaa6d8d1ce17b"
Oct 31 13:44:22.127600 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:44:22.127288    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761ebd16829fb399522f86d68a5bbdff757a5961e5ae32f5ddcaaa6d8d1ce17b\": container with ID starting with 761ebd16829fb399522f86d68a5bbdff757a5961e5ae32f5ddcaaa6d8d1ce17b not found: ID does not exist" containerID="761ebd16829fb399522f86d68a5bbdff757a5961e5ae32f5ddcaaa6d8d1ce17b"
Oct 31 13:44:22.127600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.127487    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761ebd16829fb399522f86d68a5bbdff757a5961e5ae32f5ddcaaa6d8d1ce17b"} err="failed to get container status \"761ebd16829fb399522f86d68a5bbdff757a5961e5ae32f5ddcaaa6d8d1ce17b\": rpc error: code = NotFound desc = could not find container \"761ebd16829fb399522f86d68a5bbdff757a5961e5ae32f5ddcaaa6d8d1ce17b\": container with ID starting with 761ebd16829fb399522f86d68a5bbdff757a5961e5ae32f5ddcaaa6d8d1ce17b not found: ID does not exist"
Oct 31 13:44:22.127600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.127512    4660 scope.go:122] "RemoveContainer" containerID="125ded9fb17944d870b98a5607223b07032c8ac8bceb93b3bd11f11a48703e57"
Oct 31 13:44:22.128976 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.128737    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-l55lt"]
Oct 31 13:44:22.145198 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.145120    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"]
Oct 31 13:44:22.160089 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.160058    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-8kpn6"]
Oct 31 13:44:22.163791 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.163769    4660 scope.go:122] "RemoveContainer" containerID="1deba25e69ae1f08e3994501aa5bc4ba375416444e71749d9fba2f7e93733caf"
Oct 31 13:44:22.166596 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.166573    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"]
Oct 31 13:44:22.171732 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.171708    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-xs6s9"]
Oct 31 13:44:22.183315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.180829    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"]
Oct 31 13:44:22.183315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.182073    4660 scope.go:122] "RemoveContainer" containerID="1deba25e69ae1f08e3994501aa5bc4ba375416444e71749d9fba2f7e93733caf"
Oct 31 13:44:22.183315 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:44:22.182474    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1deba25e69ae1f08e3994501aa5bc4ba375416444e71749d9fba2f7e93733caf\": container with ID starting with 1deba25e69ae1f08e3994501aa5bc4ba375416444e71749d9fba2f7e93733caf not found: ID does not exist" containerID="1deba25e69ae1f08e3994501aa5bc4ba375416444e71749d9fba2f7e93733caf"
Oct 31 13:44:22.183315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.182501    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1deba25e69ae1f08e3994501aa5bc4ba375416444e71749d9fba2f7e93733caf"} err="failed to get container status \"1deba25e69ae1f08e3994501aa5bc4ba375416444e71749d9fba2f7e93733caf\": rpc error: code = NotFound desc = could not find container \"1deba25e69ae1f08e3994501aa5bc4ba375416444e71749d9fba2f7e93733caf\": container with ID starting with 1deba25e69ae1f08e3994501aa5bc4ba375416444e71749d9fba2f7e93733caf not found: ID does not exist"
Oct 31 13:44:22.186491 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.186468    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-45dcg"]
Oct 31 13:44:22.549371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.549239    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" path="/var/lib/kubelet/pods/16adae6b-a455-42f5-a51b-f9c018eec1cc/volumes"
Oct 31 13:44:22.549865 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.549799    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" path="/var/lib/kubelet/pods/5833e6de-e1f1-425c-865f-9293c4396f1e/volumes"
Oct 31 13:44:22.550470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.550450    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c132a2a-1e7e-4858-80fb-376083a5b0f7" path="/var/lib/kubelet/pods/5c132a2a-1e7e-4858-80fb-376083a5b0f7/volumes"
Oct 31 13:44:22.551445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.551420    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8906fc94-2d02-49d6-8ab0-c578e4b7391a" path="/var/lib/kubelet/pods/8906fc94-2d02-49d6-8ab0-c578e4b7391a/volumes"
Oct 31 13:44:22.551985 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:22.551967    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd64e87-c152-497f-9460-075f624e4157" path="/var/lib/kubelet/pods/8dd64e87-c152-497f-9460-075f624e4157/volumes"
Oct 31 13:44:23.185557 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:23.185520    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"]
Oct 31 13:44:23.209018 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:23.208985    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69bdf5-5kls2"]
Oct 31 13:44:24.529781 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:24.529752    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d374cdd2-4367-4745-8290-33a05145a48d" path="/var/lib/kubelet/pods/d374cdd2-4367-4745-8290-33a05145a48d/volumes"
Oct 31 13:44:50.758003 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:44:50.757959    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:44:55.775437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:55.775400    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"]
Oct 31 13:44:58.791548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:58.785053    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"]
Oct 31 13:44:58.791548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:58.785288    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7" podUID="5d7f58a2-4f5a-4100-baeb-3d0664b21972" containerName="maas-api" containerID="cri-o://c18a2fcee4b30525346897dc9c44f9022c4e78f2e750492a259c4c1d7b620c52" gracePeriod=30
Oct 31 13:44:59.464772 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.454355    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:44:59.537122 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.537083    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d7f58a2-4f5a-4100-baeb-3d0664b21972-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7f58a2-4f5a-4100-baeb-3d0664b21972-openshift-service-ca\") pod \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\" (UID: \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\") "
Oct 31 13:44:59.537347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.537148    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7f58a2-4f5a-4100-baeb-3d0664b21972-kube-api-access-kqwgx\" (UniqueName: \"kubernetes.io/projected/5d7f58a2-4f5a-4100-baeb-3d0664b21972-kube-api-access-kqwgx\") pod \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\" (UID: \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\") "
Oct 31 13:44:59.537347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.537188    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d7f58a2-4f5a-4100-baeb-3d0664b21972-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7f58a2-4f5a-4100-baeb-3d0664b21972-metrics-certs\") pod \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\" (UID: \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\") "
Oct 31 13:44:59.537347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.537223    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d7f58a2-4f5a-4100-baeb-3d0664b21972-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7f58a2-4f5a-4100-baeb-3d0664b21972-maas-api-tls\") pod \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\" (UID: \"5d7f58a2-4f5a-4100-baeb-3d0664b21972\") "
Oct 31 13:44:59.538568 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.538540    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7f58a2-4f5a-4100-baeb-3d0664b21972-openshift-service-ca" pod "5d7f58a2-4f5a-4100-baeb-3d0664b21972" (UID: "5d7f58a2-4f5a-4100-baeb-3d0664b21972"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:59.540505 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.540449    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7f58a2-4f5a-4100-baeb-3d0664b21972-maas-api-tls" pod "5d7f58a2-4f5a-4100-baeb-3d0664b21972" (UID: "5d7f58a2-4f5a-4100-baeb-3d0664b21972"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:44:59.542674 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.542645    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7f58a2-4f5a-4100-baeb-3d0664b21972-kube-api-access-kqwgx" pod "5d7f58a2-4f5a-4100-baeb-3d0664b21972" (UID: "5d7f58a2-4f5a-4100-baeb-3d0664b21972"). InnerVolumeSpecName "kube-api-access-kqwgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:59.543719 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.543690    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7f58a2-4f5a-4100-baeb-3d0664b21972-metrics-certs" pod "5d7f58a2-4f5a-4100-baeb-3d0664b21972" (UID: "5d7f58a2-4f5a-4100-baeb-3d0664b21972"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:44:59.645091 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.639886    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7f58a2-4f5a-4100-baeb-3d0664b21972-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:59.645091 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.639937    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqwgx\" (UniqueName: \"kubernetes.io/projected/5d7f58a2-4f5a-4100-baeb-3d0664b21972-kube-api-access-kqwgx\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:59.645091 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.639953    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7f58a2-4f5a-4100-baeb-3d0664b21972-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:59.645091 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.639967    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7f58a2-4f5a-4100-baeb-3d0664b21972-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:59.657171 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.657132    4660 generic.go:356] "Generic (PLEG): container finished" podID="5d7f58a2-4f5a-4100-baeb-3d0664b21972" containerID="c18a2fcee4b30525346897dc9c44f9022c4e78f2e750492a259c4c1d7b620c52" exitCode=0
Oct 31 13:44:59.657368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.657191    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"
Oct 31 13:44:59.657368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.657227    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7" event={"ID":"5d7f58a2-4f5a-4100-baeb-3d0664b21972","Type":"ContainerDied","Data":"c18a2fcee4b30525346897dc9c44f9022c4e78f2e750492a259c4c1d7b620c52"}
Oct 31 13:44:59.657455 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.657378    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7" event={"ID":"5d7f58a2-4f5a-4100-baeb-3d0664b21972","Type":"ContainerDied","Data":"ea39e20d508dd6a2bb9e42e97886be8a2c5a8fd5e0cc11fbc14c94b1363442e0"}
Oct 31 13:44:59.657455 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.657415    4660 scope.go:122] "RemoveContainer" containerID="c18a2fcee4b30525346897dc9c44f9022c4e78f2e750492a259c4c1d7b620c52"
Oct 31 13:44:59.676455 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.676425    4660 scope.go:122] "RemoveContainer" containerID="c18a2fcee4b30525346897dc9c44f9022c4e78f2e750492a259c4c1d7b620c52"
Oct 31 13:44:59.677086 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:44:59.676896    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c18a2fcee4b30525346897dc9c44f9022c4e78f2e750492a259c4c1d7b620c52\": container with ID starting with c18a2fcee4b30525346897dc9c44f9022c4e78f2e750492a259c4c1d7b620c52 not found: ID does not exist" containerID="c18a2fcee4b30525346897dc9c44f9022c4e78f2e750492a259c4c1d7b620c52"
Oct 31 13:44:59.677086 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.676933    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c18a2fcee4b30525346897dc9c44f9022c4e78f2e750492a259c4c1d7b620c52"} err="failed to get container status \"c18a2fcee4b30525346897dc9c44f9022c4e78f2e750492a259c4c1d7b620c52\": rpc error: code = NotFound desc = could not find container \"c18a2fcee4b30525346897dc9c44f9022c4e78f2e750492a259c4c1d7b620c52\": container with ID starting with c18a2fcee4b30525346897dc9c44f9022c4e78f2e750492a259c4c1d7b620c52 not found: ID does not exist"
Oct 31 13:44:59.703890 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.703846    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"]
Oct 31 13:44:59.719818 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:44:59.719788    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d4b52a-7bcdd86ddc-jl4m7"]
Oct 31 13:45:00.190491 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.190056    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v"]
Oct 31 13:45:00.191072 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191046    4660 cpu_manager.go:414] "containerMap: removing container" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy"
Oct 31 13:45:00.191151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191074    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy"
Oct 31 13:45:00.191151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191101    4660 cpu_manager.go:414] "containerMap: removing container" podUID="8906fc94-2d02-49d6-8ab0-c578e4b7391a" containerName="istio-proxy"
Oct 31 13:45:00.191151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191110    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8906fc94-2d02-49d6-8ab0-c578e4b7391a" containerName="istio-proxy"
Oct 31 13:45:00.191151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191128    4660 cpu_manager.go:414] "containerMap: removing container" podUID="5d7f58a2-4f5a-4100-baeb-3d0664b21972" containerName="maas-api"
Oct 31 13:45:00.191151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191137    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7f58a2-4f5a-4100-baeb-3d0664b21972" containerName="maas-api"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191160    4660 cpu_manager.go:414] "containerMap: removing container" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191168    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191181    4660 cpu_manager.go:414] "containerMap: removing container" podUID="d374cdd2-4367-4745-8290-33a05145a48d" containerName="revoke-keys"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191189    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d374cdd2-4367-4745-8290-33a05145a48d" containerName="revoke-keys"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191196    4660 cpu_manager.go:414] "containerMap: removing container" podUID="e003dfad-238c-47f3-aa19-696ca2c1264a" containerName="main"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191201    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e003dfad-238c-47f3-aa19-696ca2c1264a" containerName="main"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191214    4660 cpu_manager.go:414] "containerMap: removing container" podUID="040f7f90-43a7-4162-8de4-d17a85cb15ca" containerName="istio-proxy"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191219    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040f7f90-43a7-4162-8de4-d17a85cb15ca" containerName="istio-proxy"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191232    4660 cpu_manager.go:414] "containerMap: removing container" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191236    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191253    4660 cpu_manager.go:414] "containerMap: removing container" podUID="5c132a2a-1e7e-4858-80fb-376083a5b0f7" containerName="istio-proxy"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191260    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c132a2a-1e7e-4858-80fb-376083a5b0f7" containerName="istio-proxy"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191282    4660 cpu_manager.go:414] "containerMap: removing container" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191286    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191352    4660 cpu_manager.go:414] "containerMap: removing container" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191359    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191485    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="16adae6b-a455-42f5-a51b-f9c018eec1cc" containerName="istio-proxy"
Oct 31 13:45:00.191490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191498    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="8906fc94-2d02-49d6-8ab0-c578e4b7391a" containerName="istio-proxy"
Oct 31 13:45:00.192180 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191509    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7f58a2-4f5a-4100-baeb-3d0664b21972" containerName="maas-api"
Oct 31 13:45:00.192180 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191517    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="2060533a-c13a-4a2e-bcd0-c904960ed76c" containerName="istio-proxy"
Oct 31 13:45:00.192180 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191524    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="d374cdd2-4367-4745-8290-33a05145a48d" containerName="revoke-keys"
Oct 31 13:45:00.192180 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191530    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="e003dfad-238c-47f3-aa19-696ca2c1264a" containerName="main"
Oct 31 13:45:00.192180 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191538    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="040f7f90-43a7-4162-8de4-d17a85cb15ca" containerName="istio-proxy"
Oct 31 13:45:00.192180 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191545    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="1daa31f8-4e70-413f-9529-0a67eef57732" containerName="istio-proxy"
Oct 31 13:45:00.192180 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191551    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c132a2a-1e7e-4858-80fb-376083a5b0f7" containerName="istio-proxy"
Oct 31 13:45:00.192180 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191559    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd64e87-c152-497f-9460-075f624e4157" containerName="istio-proxy"
Oct 31 13:45:00.192180 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.191568    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="5833e6de-e1f1-425c-865f-9293c4396f1e" containerName="istio-proxy"
Oct 31 13:45:00.234360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.224122    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v"
Oct 31 13:45:00.234360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.229193    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld"]
Oct 31 13:45:00.281178 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.280594    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld"
Oct 31 13:45:00.281394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.280500    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v"]
Oct 31 13:45:00.281537 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.281524    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld"]
Oct 31 13:45:00.285562 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.285531    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7"]
Oct 31 13:45:00.291976 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.291470    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4"]
Oct 31 13:45:00.291976 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.291574    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7"
Oct 31 13:45:00.300858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.298762    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:45:00.301723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.301698    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4"
Oct 31 13:45:00.314511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.314487    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7"]
Oct 31 13:45:00.351694 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.351669    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl"]
Oct 31 13:45:00.358537 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.358511    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4"]
Oct 31 13:45:00.358664 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.358622    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl"
Oct 31 13:45:00.370728 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.370699    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m"]
Oct 31 13:45:00.371949 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.371918    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61588bc9-c966-4ea3-9a38-750555e43840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld\" (UID: \"61588bc9-c966-4ea3-9a38-750555e43840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld"
Oct 31 13:45:00.372064 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.372026    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kk2x\" (UniqueName: \"kubernetes.io/projected/61588bc9-c966-4ea3-9a38-750555e43840-kube-api-access-8kk2x\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld\" (UID: \"61588bc9-c966-4ea3-9a38-750555e43840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld"
Oct 31 13:45:00.372161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.372141    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8e87df-0d34-443a-ba4a-11f647aed571-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7\" (UID: \"ad8e87df-0d34-443a-ba4a-11f647aed571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7"
Oct 31 13:45:00.372254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.372236    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb4vn\" (UniqueName: \"kubernetes.io/projected/a37b859d-6ef3-4e77-9459-47939791f9c6-kube-api-access-kb4vn\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v\" (UID: \"a37b859d-6ef3-4e77-9459-47939791f9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v"
Oct 31 13:45:00.372379 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.372361    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37b859d-6ef3-4e77-9459-47939791f9c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v\" (UID: \"a37b859d-6ef3-4e77-9459-47939791f9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v"
Oct 31 13:45:00.372472 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.372456    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xmvq\" (UniqueName: \"kubernetes.io/projected/ad8e87df-0d34-443a-ba4a-11f647aed571-kube-api-access-6xmvq\") pod \"maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7\" (UID: \"ad8e87df-0d34-443a-ba4a-11f647aed571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7"
Oct 31 13:45:00.378777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.378756    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m"
Oct 31 13:45:00.395513 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.395486    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl"]
Oct 31 13:45:00.407687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.407660    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m"]
Oct 31 13:45:00.473813 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.473718    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftwfw\" (UniqueName: \"kubernetes.io/projected/bb79fc7d-a4bb-4e66-b320-eeff445ff5c0-kube-api-access-ftwfw\") pod \"maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl\" (UID: \"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl"
Oct 31 13:45:00.474492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.473848    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37b859d-6ef3-4e77-9459-47939791f9c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v\" (UID: \"a37b859d-6ef3-4e77-9459-47939791f9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v"
Oct 31 13:45:00.474492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.473927    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa08b46-4029-44a9-9c58-1ae42760deb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m\" (UID: \"afa08b46-4029-44a9-9c58-1ae42760deb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m"
Oct 31 13:45:00.474492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.473987    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xmvq\" (UniqueName: \"kubernetes.io/projected/ad8e87df-0d34-443a-ba4a-11f647aed571-kube-api-access-6xmvq\") pod \"maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7\" (UID: \"ad8e87df-0d34-443a-ba4a-11f647aed571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7"
Oct 31 13:45:00.474492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.474184    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61588bc9-c966-4ea3-9a38-750555e43840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld\" (UID: \"61588bc9-c966-4ea3-9a38-750555e43840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld"
Oct 31 13:45:00.474492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.474266    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94082533-524e-4201-ae61-c73877895d65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4\" (UID: \"94082533-524e-4201-ae61-c73877895d65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4"
Oct 31 13:45:00.474492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.474330    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kk2x\" (UniqueName: \"kubernetes.io/projected/61588bc9-c966-4ea3-9a38-750555e43840-kube-api-access-8kk2x\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld\" (UID: \"61588bc9-c966-4ea3-9a38-750555e43840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld"
Oct 31 13:45:00.474492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.474375    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb79fc7d-a4bb-4e66-b320-eeff445ff5c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl\" (UID: \"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl"
Oct 31 13:45:00.474492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.474422    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8e87df-0d34-443a-ba4a-11f647aed571-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7\" (UID: \"ad8e87df-0d34-443a-ba4a-11f647aed571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7"
Oct 31 13:45:00.474492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.474472    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvhs7\" (UniqueName: \"kubernetes.io/projected/afa08b46-4029-44a9-9c58-1ae42760deb7-kube-api-access-lvhs7\") pod \"maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m\" (UID: \"afa08b46-4029-44a9-9c58-1ae42760deb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m"
Oct 31 13:45:00.474876 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.474605    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37b859d-6ef3-4e77-9459-47939791f9c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v\" (UID: \"a37b859d-6ef3-4e77-9459-47939791f9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v"
Oct 31 13:45:00.474876 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.474664    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb4vn\" (UniqueName: \"kubernetes.io/projected/a37b859d-6ef3-4e77-9459-47939791f9c6-kube-api-access-kb4vn\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v\" (UID: \"a37b859d-6ef3-4e77-9459-47939791f9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v"
Oct 31 13:45:00.474876 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.474696    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcf7j\" (UniqueName: \"kubernetes.io/projected/94082533-524e-4201-ae61-c73877895d65-kube-api-access-wcf7j\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4\" (UID: \"94082533-524e-4201-ae61-c73877895d65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4"
Oct 31 13:45:00.475067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.475042    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8e87df-0d34-443a-ba4a-11f647aed571-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7\" (UID: \"ad8e87df-0d34-443a-ba4a-11f647aed571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7"
Oct 31 13:45:00.475506 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.475487    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61588bc9-c966-4ea3-9a38-750555e43840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld\" (UID: \"61588bc9-c966-4ea3-9a38-750555e43840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld"
Oct 31 13:45:00.487428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.487400    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xmvq\" (UniqueName: \"kubernetes.io/projected/ad8e87df-0d34-443a-ba4a-11f647aed571-kube-api-access-6xmvq\") pod \"maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7\" (UID: \"ad8e87df-0d34-443a-ba4a-11f647aed571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7"
Oct 31 13:45:00.488341 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.487771    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb4vn\" (UniqueName: \"kubernetes.io/projected/a37b859d-6ef3-4e77-9459-47939791f9c6-kube-api-access-kb4vn\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v\" (UID: \"a37b859d-6ef3-4e77-9459-47939791f9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v"
Oct 31 13:45:00.488341 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.488237    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kk2x\" (UniqueName: \"kubernetes.io/projected/61588bc9-c966-4ea3-9a38-750555e43840-kube-api-access-8kk2x\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld\" (UID: \"61588bc9-c966-4ea3-9a38-750555e43840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld"
Oct 31 13:45:00.534607 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.534394    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d7f58a2-4f5a-4100-baeb-3d0664b21972" path="/var/lib/kubelet/pods/5d7f58a2-4f5a-4100-baeb-3d0664b21972/volumes"
Oct 31 13:45:00.576569 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.576533    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94082533-524e-4201-ae61-c73877895d65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4\" (UID: \"94082533-524e-4201-ae61-c73877895d65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4"
Oct 31 13:45:00.576752 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.576602    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb79fc7d-a4bb-4e66-b320-eeff445ff5c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl\" (UID: \"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl"
Oct 31 13:45:00.576752 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.576662    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvhs7\" (UniqueName: \"kubernetes.io/projected/afa08b46-4029-44a9-9c58-1ae42760deb7-kube-api-access-lvhs7\") pod \"maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m\" (UID: \"afa08b46-4029-44a9-9c58-1ae42760deb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m"
Oct 31 13:45:00.576752 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.576732    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcf7j\" (UniqueName: \"kubernetes.io/projected/94082533-524e-4201-ae61-c73877895d65-kube-api-access-wcf7j\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4\" (UID: \"94082533-524e-4201-ae61-c73877895d65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4"
Oct 31 13:45:00.576928 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.576796    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwfw\" (UniqueName: \"kubernetes.io/projected/bb79fc7d-a4bb-4e66-b320-eeff445ff5c0-kube-api-access-ftwfw\") pod \"maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl\" (UID: \"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl"
Oct 31 13:45:00.576928 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.576888    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa08b46-4029-44a9-9c58-1ae42760deb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m\" (UID: \"afa08b46-4029-44a9-9c58-1ae42760deb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m"
Oct 31 13:45:00.577409 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.577385    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb79fc7d-a4bb-4e66-b320-eeff445ff5c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl\" (UID: \"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl"
Oct 31 13:45:00.577521 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.577498    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94082533-524e-4201-ae61-c73877895d65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4\" (UID: \"94082533-524e-4201-ae61-c73877895d65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4"
Oct 31 13:45:00.577764 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.577743    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa08b46-4029-44a9-9c58-1ae42760deb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m\" (UID: \"afa08b46-4029-44a9-9c58-1ae42760deb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m"
Oct 31 13:45:00.591007 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.590972    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcf7j\" (UniqueName: \"kubernetes.io/projected/94082533-524e-4201-ae61-c73877895d65-kube-api-access-wcf7j\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4\" (UID: \"94082533-524e-4201-ae61-c73877895d65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4"
Oct 31 13:45:00.591139 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.590971    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvhs7\" (UniqueName: \"kubernetes.io/projected/afa08b46-4029-44a9-9c58-1ae42760deb7-kube-api-access-lvhs7\") pod \"maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m\" (UID: \"afa08b46-4029-44a9-9c58-1ae42760deb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m"
Oct 31 13:45:00.596700 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.596672    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftwfw\" (UniqueName: \"kubernetes.io/projected/bb79fc7d-a4bb-4e66-b320-eeff445ff5c0-kube-api-access-ftwfw\") pod \"maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl\" (UID: \"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl"
Oct 31 13:45:00.603682 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.603658    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v"
Oct 31 13:45:00.644315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.642791    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld"
Oct 31 13:45:00.647885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.647857    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7"
Oct 31 13:45:00.673628 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.672998    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4"
Oct 31 13:45:00.735698 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.706835    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl"
Oct 31 13:45:00.735698 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:00.718508    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m"
Oct 31 13:45:01.366320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:01.365201    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v"]
Oct 31 13:45:01.378959 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:45:01.378919    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37b859d_6ef3_4e77_9459_47939791f9c6.slice/crio-38727c0981767885ea4013d6f007f2f6a713f24d23403f3970565cbb2120bde7 WatchSource:0}: Error finding container 38727c0981767885ea4013d6f007f2f6a713f24d23403f3970565cbb2120bde7: Status 404 returned error can't find the container with id 38727c0981767885ea4013d6f007f2f6a713f24d23403f3970565cbb2120bde7
Oct 31 13:45:01.583073 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:01.583041    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld"]
Oct 31 13:45:01.705191 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:01.705149    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4"]
Oct 31 13:45:01.721786 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:01.721752    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v" event={"ID":"a37b859d-6ef3-4e77-9459-47939791f9c6","Type":"ContainerStarted","Data":"38727c0981767885ea4013d6f007f2f6a713f24d23403f3970565cbb2120bde7"}
Oct 31 13:45:01.723991 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:01.723950    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld" event={"ID":"61588bc9-c966-4ea3-9a38-750555e43840","Type":"ContainerStarted","Data":"a3caa2e31591024dd2bf42711d83a31be3c56534714b840d5a34cc2875304bef"}
Oct 31 13:45:01.729743 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:01.729718    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7"]
Oct 31 13:45:01.730053 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:45:01.730032    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8e87df_0d34_443a_ba4a_11f647aed571.slice/crio-78b701eb72c5e75a7f5d4d27ad89146bd3a59e2b916ab15a3d8c93de3c6eaffa WatchSource:0}: Error finding container 78b701eb72c5e75a7f5d4d27ad89146bd3a59e2b916ab15a3d8c93de3c6eaffa: Status 404 returned error can't find the container with id 78b701eb72c5e75a7f5d4d27ad89146bd3a59e2b916ab15a3d8c93de3c6eaffa
Oct 31 13:45:01.814842 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:45:01.814791    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb79fc7d_a4bb_4e66_b320_eeff445ff5c0.slice/crio-3e88181c2d9a14af0b5f01cc22530b89f2c4a01b77adcf8aca94d05903ca5b9b WatchSource:0}: Error finding container 3e88181c2d9a14af0b5f01cc22530b89f2c4a01b77adcf8aca94d05903ca5b9b: Status 404 returned error can't find the container with id 3e88181c2d9a14af0b5f01cc22530b89f2c4a01b77adcf8aca94d05903ca5b9b
Oct 31 13:45:01.817800 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:01.817779    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl"]
Oct 31 13:45:01.825967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:01.825938    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m"]
Oct 31 13:45:02.739136 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.739049    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4" event={"ID":"94082533-524e-4201-ae61-c73877895d65","Type":"ContainerStarted","Data":"e2f721c0a653bdb5721c4fb536f6f6abbb7e63ae26ae7548a1ec03b20b9dd544"}
Oct 31 13:45:02.739136 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.739115    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4" event={"ID":"94082533-524e-4201-ae61-c73877895d65","Type":"ContainerStarted","Data":"6201b3a71f0228f0971f1dd3bfe6d75ba4d64162010f23c82ec0eda419d8afe8"}
Oct 31 13:45:02.741347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.741320    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl" event={"ID":"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0","Type":"ContainerStarted","Data":"9b1633f8504bd09d589dd1cd310cd07a4d4becd1a24870bf25ccd57852c40f31"}
Oct 31 13:45:02.741487 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.741362    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl" event={"ID":"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0","Type":"ContainerStarted","Data":"3e88181c2d9a14af0b5f01cc22530b89f2c4a01b77adcf8aca94d05903ca5b9b"}
Oct 31 13:45:02.743271 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.743242    4660 generic.go:356] "Generic (PLEG): container finished" podID="a37b859d-6ef3-4e77-9459-47939791f9c6" containerID="c974bdbc560e78100d258a687fc0d37035c9ee9944a692bb68d0916c8fb0ddf3" exitCode=60
Oct 31 13:45:02.743392 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.743338    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v" event={"ID":"a37b859d-6ef3-4e77-9459-47939791f9c6","Type":"ContainerDied","Data":"c974bdbc560e78100d258a687fc0d37035c9ee9944a692bb68d0916c8fb0ddf3"}
Oct 31 13:45:02.743642 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.743622    4660 scope.go:122] "RemoveContainer" containerID="c974bdbc560e78100d258a687fc0d37035c9ee9944a692bb68d0916c8fb0ddf3"
Oct 31 13:45:02.745434 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.745413    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m" event={"ID":"afa08b46-4029-44a9-9c58-1ae42760deb7","Type":"ContainerStarted","Data":"d24296f5f1b863ff6702f1aff8c41799919c71941a09c3addca1358dbff46190"}
Oct 31 13:45:02.745522 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.745448    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m" event={"ID":"afa08b46-4029-44a9-9c58-1ae42760deb7","Type":"ContainerStarted","Data":"3406c983f14071b3e47dbf644826c2e498770af1989af2d98cb42253de77875f"}
Oct 31 13:45:02.747622 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.747592    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld" event={"ID":"61588bc9-c966-4ea3-9a38-750555e43840","Type":"ContainerStarted","Data":"c7412d2de3abe783007b78844428119a37e23b4c4380207e3243493d6665aac5"}
Oct 31 13:45:02.749370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.749346    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7" event={"ID":"ad8e87df-0d34-443a-ba4a-11f647aed571","Type":"ContainerStarted","Data":"5bb3b7bc3055f0f7d9149ba612b7d28300b30814f607e8e1345a40238205190b"}
Oct 31 13:45:02.749461 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.749391    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7" event={"ID":"ad8e87df-0d34-443a-ba4a-11f647aed571","Type":"ContainerStarted","Data":"78b701eb72c5e75a7f5d4d27ad89146bd3a59e2b916ab15a3d8c93de3c6eaffa"}
Oct 31 13:45:02.761194 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.761151    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4" podStartSLOduration=2.761138948 podStartE2EDuration="2.761138948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:02.759598708 +0000 UTC m=+2502.378257872" watchObservedRunningTime="2026-10-31 13:45:02.761138948 +0000 UTC m=+2502.379798111"
Oct 31 13:45:02.782080 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.782037    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m" podStartSLOduration=2.782024787 podStartE2EDuration="2.782024787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:02.781431453 +0000 UTC m=+2502.400090617" watchObservedRunningTime="2026-10-31 13:45:02.782024787 +0000 UTC m=+2502.400683954"
Oct 31 13:45:02.830806 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.830748    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl" podStartSLOduration=2.830730386 podStartE2EDuration="2.830730386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:02.830234474 +0000 UTC m=+2502.448893639" watchObservedRunningTime="2026-10-31 13:45:02.830730386 +0000 UTC m=+2502.449389552"
Oct 31 13:45:02.857610 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.857560    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld" podStartSLOduration=2.857547148 podStartE2EDuration="2.857547148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:02.852239951 +0000 UTC m=+2502.470899108" watchObservedRunningTime="2026-10-31 13:45:02.857547148 +0000 UTC m=+2502.476206354"
Oct 31 13:45:02.947204 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:02.947151    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7" podStartSLOduration=2.947136454 podStartE2EDuration="2.947136454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:02.943913924 +0000 UTC m=+2502.562573088" watchObservedRunningTime="2026-10-31 13:45:02.947136454 +0000 UTC m=+2502.565795617"
Oct 31 13:45:03.759699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.759666    4660 generic.go:356] "Generic (PLEG): container finished" podID="94082533-524e-4201-ae61-c73877895d65" containerID="e2f721c0a653bdb5721c4fb536f6f6abbb7e63ae26ae7548a1ec03b20b9dd544" exitCode=60
Oct 31 13:45:03.760108 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.759752    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4" event={"ID":"94082533-524e-4201-ae61-c73877895d65","Type":"ContainerDied","Data":"e2f721c0a653bdb5721c4fb536f6f6abbb7e63ae26ae7548a1ec03b20b9dd544"}
Oct 31 13:45:03.760108 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.760059    4660 scope.go:122] "RemoveContainer" containerID="e2f721c0a653bdb5721c4fb536f6f6abbb7e63ae26ae7548a1ec03b20b9dd544"
Oct 31 13:45:03.761712 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.761683    4660 generic.go:356] "Generic (PLEG): container finished" podID="bb79fc7d-a4bb-4e66-b320-eeff445ff5c0" containerID="9b1633f8504bd09d589dd1cd310cd07a4d4becd1a24870bf25ccd57852c40f31" exitCode=60
Oct 31 13:45:03.761822 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.761762    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl" event={"ID":"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0","Type":"ContainerDied","Data":"9b1633f8504bd09d589dd1cd310cd07a4d4becd1a24870bf25ccd57852c40f31"}
Oct 31 13:45:03.762060 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.762046    4660 scope.go:122] "RemoveContainer" containerID="9b1633f8504bd09d589dd1cd310cd07a4d4becd1a24870bf25ccd57852c40f31"
Oct 31 13:45:03.764274 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.764197    4660 generic.go:356] "Generic (PLEG): container finished" podID="a37b859d-6ef3-4e77-9459-47939791f9c6" containerID="a88b5de348a86e548b6a415a58ac0ad6ea2ff4d992dce31240d9d5164be510aa" exitCode=60
Oct 31 13:45:03.764399 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.764270    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v" event={"ID":"a37b859d-6ef3-4e77-9459-47939791f9c6","Type":"ContainerDied","Data":"a88b5de348a86e548b6a415a58ac0ad6ea2ff4d992dce31240d9d5164be510aa"}
Oct 31 13:45:03.764399 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.764365    4660 scope.go:122] "RemoveContainer" containerID="c974bdbc560e78100d258a687fc0d37035c9ee9944a692bb68d0916c8fb0ddf3"
Oct 31 13:45:03.764660 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.764638    4660 scope.go:122] "RemoveContainer" containerID="a88b5de348a86e548b6a415a58ac0ad6ea2ff4d992dce31240d9d5164be510aa"
Oct 31 13:45:03.765891 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:03.765870    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v_odh-ai-gateway-infra(a37b859d-6ef3-4e77-9459-47939791f9c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v" podUID="a37b859d-6ef3-4e77-9459-47939791f9c6"
Oct 31 13:45:03.766436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.766413    4660 generic.go:356] "Generic (PLEG): container finished" podID="afa08b46-4029-44a9-9c58-1ae42760deb7" containerID="d24296f5f1b863ff6702f1aff8c41799919c71941a09c3addca1358dbff46190" exitCode=60
Oct 31 13:45:03.766497 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.766484    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m" event={"ID":"afa08b46-4029-44a9-9c58-1ae42760deb7","Type":"ContainerDied","Data":"d24296f5f1b863ff6702f1aff8c41799919c71941a09c3addca1358dbff46190"}
Oct 31 13:45:03.766675 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.766655    4660 scope.go:122] "RemoveContainer" containerID="d24296f5f1b863ff6702f1aff8c41799919c71941a09c3addca1358dbff46190"
Oct 31 13:45:03.768330 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.768286    4660 generic.go:356] "Generic (PLEG): container finished" podID="61588bc9-c966-4ea3-9a38-750555e43840" containerID="c7412d2de3abe783007b78844428119a37e23b4c4380207e3243493d6665aac5" exitCode=60
Oct 31 13:45:03.768430 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.768331    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld" event={"ID":"61588bc9-c966-4ea3-9a38-750555e43840","Type":"ContainerDied","Data":"c7412d2de3abe783007b78844428119a37e23b4c4380207e3243493d6665aac5"}
Oct 31 13:45:03.768598 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.768582    4660 scope.go:122] "RemoveContainer" containerID="c7412d2de3abe783007b78844428119a37e23b4c4380207e3243493d6665aac5"
Oct 31 13:45:03.770383 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.770208    4660 generic.go:356] "Generic (PLEG): container finished" podID="ad8e87df-0d34-443a-ba4a-11f647aed571" containerID="5bb3b7bc3055f0f7d9149ba612b7d28300b30814f607e8e1345a40238205190b" exitCode=60
Oct 31 13:45:03.770383 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.770263    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7" event={"ID":"ad8e87df-0d34-443a-ba4a-11f647aed571","Type":"ContainerDied","Data":"5bb3b7bc3055f0f7d9149ba612b7d28300b30814f607e8e1345a40238205190b"}
Oct 31 13:45:03.770559 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:03.770536    4660 scope.go:122] "RemoveContainer" containerID="5bb3b7bc3055f0f7d9149ba612b7d28300b30814f607e8e1345a40238205190b"
Oct 31 13:45:04.783904 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:04.783861    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4" event={"ID":"94082533-524e-4201-ae61-c73877895d65","Type":"ContainerStarted","Data":"6139cca17ff0f165e73697af5dedcd570542081b82ec5f7ce6d2a0f7e27ca166"}
Oct 31 13:45:04.786806 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:04.786772    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl" event={"ID":"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0","Type":"ContainerStarted","Data":"e372f33e9356f3238d2b8956d85eaab5fec4571662993d2242c3c7d7db0f5837"}
Oct 31 13:45:04.791049 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:04.791017    4660 scope.go:122] "RemoveContainer" containerID="a88b5de348a86e548b6a415a58ac0ad6ea2ff4d992dce31240d9d5164be510aa"
Oct 31 13:45:04.791510 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:04.791483    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v_odh-ai-gateway-infra(a37b859d-6ef3-4e77-9459-47939791f9c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v" podUID="a37b859d-6ef3-4e77-9459-47939791f9c6"
Oct 31 13:45:04.795341 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:04.795312    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m" event={"ID":"afa08b46-4029-44a9-9c58-1ae42760deb7","Type":"ContainerStarted","Data":"b7ff63489c0205eb6359942724975fb631e5190626a485e6ced122df1cee73fd"}
Oct 31 13:45:04.797872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:04.797848    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld" event={"ID":"61588bc9-c966-4ea3-9a38-750555e43840","Type":"ContainerStarted","Data":"54c7fb71e8ab2aa9d787f1bd1f8db6b13ffd44f5de83c4a2153a11078319dea3"}
Oct 31 13:45:04.800775 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:04.800744    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7" event={"ID":"ad8e87df-0d34-443a-ba4a-11f647aed571","Type":"ContainerStarted","Data":"4d182d2f32fdf3ffb619062eead50b88aa92132ba5659472493952aedaf749a8"}
Oct 31 13:45:05.815030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.814988    4660 generic.go:356] "Generic (PLEG): container finished" podID="afa08b46-4029-44a9-9c58-1ae42760deb7" containerID="b7ff63489c0205eb6359942724975fb631e5190626a485e6ced122df1cee73fd" exitCode=60
Oct 31 13:45:05.815597 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.815026    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m" event={"ID":"afa08b46-4029-44a9-9c58-1ae42760deb7","Type":"ContainerDied","Data":"b7ff63489c0205eb6359942724975fb631e5190626a485e6ced122df1cee73fd"}
Oct 31 13:45:05.815597 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.815100    4660 scope.go:122] "RemoveContainer" containerID="d24296f5f1b863ff6702f1aff8c41799919c71941a09c3addca1358dbff46190"
Oct 31 13:45:05.816001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.815977    4660 scope.go:122] "RemoveContainer" containerID="b7ff63489c0205eb6359942724975fb631e5190626a485e6ced122df1cee73fd"
Oct 31 13:45:05.816718 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:05.816438    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m_odh-ai-gateway-infra(afa08b46-4029-44a9-9c58-1ae42760deb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m" podUID="afa08b46-4029-44a9-9c58-1ae42760deb7"
Oct 31 13:45:05.820710 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.820567    4660 generic.go:356] "Generic (PLEG): container finished" podID="61588bc9-c966-4ea3-9a38-750555e43840" containerID="54c7fb71e8ab2aa9d787f1bd1f8db6b13ffd44f5de83c4a2153a11078319dea3" exitCode=60
Oct 31 13:45:05.820809 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.820749    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld" event={"ID":"61588bc9-c966-4ea3-9a38-750555e43840","Type":"ContainerDied","Data":"54c7fb71e8ab2aa9d787f1bd1f8db6b13ffd44f5de83c4a2153a11078319dea3"}
Oct 31 13:45:05.821263 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.821236    4660 scope.go:122] "RemoveContainer" containerID="54c7fb71e8ab2aa9d787f1bd1f8db6b13ffd44f5de83c4a2153a11078319dea3"
Oct 31 13:45:05.821677 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:05.821651    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld_odh-ai-gateway-infra(61588bc9-c966-4ea3-9a38-750555e43840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld" podUID="61588bc9-c966-4ea3-9a38-750555e43840"
Oct 31 13:45:05.825217 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.825185    4660 generic.go:356] "Generic (PLEG): container finished" podID="ad8e87df-0d34-443a-ba4a-11f647aed571" containerID="4d182d2f32fdf3ffb619062eead50b88aa92132ba5659472493952aedaf749a8" exitCode=60
Oct 31 13:45:05.825531 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.825502    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7" event={"ID":"ad8e87df-0d34-443a-ba4a-11f647aed571","Type":"ContainerDied","Data":"4d182d2f32fdf3ffb619062eead50b88aa92132ba5659472493952aedaf749a8"}
Oct 31 13:45:05.825718 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.825700    4660 scope.go:122] "RemoveContainer" containerID="4d182d2f32fdf3ffb619062eead50b88aa92132ba5659472493952aedaf749a8"
Oct 31 13:45:05.826082 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:05.826057    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7_odh-ai-gateway-infra(ad8e87df-0d34-443a-ba4a-11f647aed571)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7" podUID="ad8e87df-0d34-443a-ba4a-11f647aed571"
Oct 31 13:45:05.828542 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.828447    4660 generic.go:356] "Generic (PLEG): container finished" podID="94082533-524e-4201-ae61-c73877895d65" containerID="6139cca17ff0f165e73697af5dedcd570542081b82ec5f7ce6d2a0f7e27ca166" exitCode=60
Oct 31 13:45:05.828683 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.828662    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4" event={"ID":"94082533-524e-4201-ae61-c73877895d65","Type":"ContainerDied","Data":"6139cca17ff0f165e73697af5dedcd570542081b82ec5f7ce6d2a0f7e27ca166"}
Oct 31 13:45:05.830188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.829839    4660 scope.go:122] "RemoveContainer" containerID="6139cca17ff0f165e73697af5dedcd570542081b82ec5f7ce6d2a0f7e27ca166"
Oct 31 13:45:05.830431 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:05.830232    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4_odh-ai-gateway-infra(94082533-524e-4201-ae61-c73877895d65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4" podUID="94082533-524e-4201-ae61-c73877895d65"
Oct 31 13:45:05.831791 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.831766    4660 generic.go:356] "Generic (PLEG): container finished" podID="bb79fc7d-a4bb-4e66-b320-eeff445ff5c0" containerID="e372f33e9356f3238d2b8956d85eaab5fec4571662993d2242c3c7d7db0f5837" exitCode=60
Oct 31 13:45:05.831791 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.831813    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl" event={"ID":"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0","Type":"ContainerDied","Data":"e372f33e9356f3238d2b8956d85eaab5fec4571662993d2242c3c7d7db0f5837"}
Oct 31 13:45:05.832154 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.832134    4660 scope.go:122] "RemoveContainer" containerID="e372f33e9356f3238d2b8956d85eaab5fec4571662993d2242c3c7d7db0f5837"
Oct 31 13:45:05.832620 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:05.832598    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl_odh-ai-gateway-infra(bb79fc7d-a4bb-4e66-b320-eeff445ff5c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl" podUID="bb79fc7d-a4bb-4e66-b320-eeff445ff5c0"
Oct 31 13:45:05.841657 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.841627    4660 scope.go:122] "RemoveContainer" containerID="c7412d2de3abe783007b78844428119a37e23b4c4380207e3243493d6665aac5"
Oct 31 13:45:05.865351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.865275    4660 scope.go:122] "RemoveContainer" containerID="5bb3b7bc3055f0f7d9149ba612b7d28300b30814f607e8e1345a40238205190b"
Oct 31 13:45:05.880181 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.880151    4660 scope.go:122] "RemoveContainer" containerID="e2f721c0a653bdb5721c4fb536f6f6abbb7e63ae26ae7548a1ec03b20b9dd544"
Oct 31 13:45:05.894877 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:05.894857    4660 scope.go:122] "RemoveContainer" containerID="9b1633f8504bd09d589dd1cd310cd07a4d4becd1a24870bf25ccd57852c40f31"
Oct 31 13:45:06.856639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:06.855882    4660 scope.go:122] "RemoveContainer" containerID="54c7fb71e8ab2aa9d787f1bd1f8db6b13ffd44f5de83c4a2153a11078319dea3"
Oct 31 13:45:06.860314 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:06.857342    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld_odh-ai-gateway-infra(61588bc9-c966-4ea3-9a38-750555e43840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld" podUID="61588bc9-c966-4ea3-9a38-750555e43840"
Oct 31 13:45:06.860314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:06.857973    4660 scope.go:122] "RemoveContainer" containerID="4d182d2f32fdf3ffb619062eead50b88aa92132ba5659472493952aedaf749a8"
Oct 31 13:45:06.860314 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:06.858547    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7_odh-ai-gateway-infra(ad8e87df-0d34-443a-ba4a-11f647aed571)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7" podUID="ad8e87df-0d34-443a-ba4a-11f647aed571"
Oct 31 13:45:06.860314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:06.860237    4660 scope.go:122] "RemoveContainer" containerID="6139cca17ff0f165e73697af5dedcd570542081b82ec5f7ce6d2a0f7e27ca166"
Oct 31 13:45:06.860769 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:06.860737    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4_odh-ai-gateway-infra(94082533-524e-4201-ae61-c73877895d65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4" podUID="94082533-524e-4201-ae61-c73877895d65"
Oct 31 13:45:06.862682 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:06.862653    4660 scope.go:122] "RemoveContainer" containerID="e372f33e9356f3238d2b8956d85eaab5fec4571662993d2242c3c7d7db0f5837"
Oct 31 13:45:06.863155 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:06.863126    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl_odh-ai-gateway-infra(bb79fc7d-a4bb-4e66-b320-eeff445ff5c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl" podUID="bb79fc7d-a4bb-4e66-b320-eeff445ff5c0"
Oct 31 13:45:06.866650 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:06.866628    4660 scope.go:122] "RemoveContainer" containerID="b7ff63489c0205eb6359942724975fb631e5190626a485e6ced122df1cee73fd"
Oct 31 13:45:06.866921 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:06.866901    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m_odh-ai-gateway-infra(afa08b46-4029-44a9-9c58-1ae42760deb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m" podUID="afa08b46-4029-44a9-9c58-1ae42760deb7"
Oct 31 13:45:10.807611 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:10.807565    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq" podUID="13349545-8422-4466-9152-bd143810b334" containerName="main" containerID="cri-o://f790a4b7f6c455619d4d9d3e7e5ac3770a55a4065698033f25c774766bd5b2e3" gracePeriod=45
Oct 31 13:45:10.954467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:10.954346    4660 generic.go:356] "Generic (PLEG): container finished" podID="13349545-8422-4466-9152-bd143810b334" containerID="f790a4b7f6c455619d4d9d3e7e5ac3770a55a4065698033f25c774766bd5b2e3" exitCode=0
Oct 31 13:45:10.954467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:10.954401    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq" event={"ID":"13349545-8422-4466-9152-bd143810b334","Type":"ContainerDied","Data":"f790a4b7f6c455619d4d9d3e7e5ac3770a55a4065698033f25c774766bd5b2e3"}
Oct 31 13:45:11.150921 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.143248    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"]
Oct 31 13:45:11.178434 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.177770    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.183579 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.182901    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-6a3533-openshift-default-dockercfg-79xkn\""
Oct 31 13:45:11.213600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.213572    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:45:11.220533 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.220507    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"]
Oct 31 13:45:11.335495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.330240    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13349545-8422-4466-9152-bd143810b334-tls-certs\" (UniqueName: \"kubernetes.io/secret/13349545-8422-4466-9152-bd143810b334-tls-certs\") pod \"13349545-8422-4466-9152-bd143810b334\" (UID: \"13349545-8422-4466-9152-bd143810b334\") "
Oct 31 13:45:11.335495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.330361    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13349545-8422-4466-9152-bd143810b334-kube-api-access-xrfq8\" (UniqueName: \"kubernetes.io/projected/13349545-8422-4466-9152-bd143810b334-kube-api-access-xrfq8\") pod \"13349545-8422-4466-9152-bd143810b334\" (UID: \"13349545-8422-4466-9152-bd143810b334\") "
Oct 31 13:45:11.335495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.330447    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-tmp-dir\") pod \"13349545-8422-4466-9152-bd143810b334\" (UID: \"13349545-8422-4466-9152-bd143810b334\") "
Oct 31 13:45:11.335495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.330587    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-model-cache\") pod \"13349545-8422-4466-9152-bd143810b334\" (UID: \"13349545-8422-4466-9152-bd143810b334\") "
Oct 31 13:45:11.335495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.330685    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-dshm\") pod \"13349545-8422-4466-9152-bd143810b334\" (UID: \"13349545-8422-4466-9152-bd143810b334\") "
Oct 31 13:45:11.335495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.330851    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-home\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-home\") pod \"13349545-8422-4466-9152-bd143810b334\" (UID: \"13349545-8422-4466-9152-bd143810b334\") "
Oct 31 13:45:11.335495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.331170    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-workload-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.335495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.331244    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-podinfo\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.365479 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.338405    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-envoy\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.365479 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.338461    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-data\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.365479 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.338650    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-token\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.365479 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.338695    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cnx4\" (UniqueName: \"kubernetes.io/projected/acc4f15d-78ea-4803-9d9d-ad445710d934-kube-api-access-8cnx4\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.365479 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.338765    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc4f15d-78ea-4803-9d9d-ad445710d934-istiod-ca-cert\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.365479 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.338833    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-workload-certs\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.365479 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.338933    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-credential-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.391380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.370212    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13349545-8422-4466-9152-bd143810b334-kube-api-access-xrfq8" pod "13349545-8422-4466-9152-bd143810b334" (UID: "13349545-8422-4466-9152-bd143810b334"). InnerVolumeSpecName "kube-api-access-xrfq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:11.391380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.370472    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-tmp-dir" pod "13349545-8422-4466-9152-bd143810b334" (UID: "13349545-8422-4466-9152-bd143810b334"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:11.391380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.370642    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-model-cache" pod "13349545-8422-4466-9152-bd143810b334" (UID: "13349545-8422-4466-9152-bd143810b334"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:11.391380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.380876    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-home" pod "13349545-8422-4466-9152-bd143810b334" (UID: "13349545-8422-4466-9152-bd143810b334"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:11.438316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.418328    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-dshm" pod "13349545-8422-4466-9152-bd143810b334" (UID: "13349545-8422-4466-9152-bd143810b334"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.445694    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-workload-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.445784    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-podinfo\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.445878    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-envoy\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.445915    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-data\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.446059    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-token\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.446105    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cnx4\" (UniqueName: \"kubernetes.io/projected/acc4f15d-78ea-4803-9d9d-ad445710d934-kube-api-access-8cnx4\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.446171    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc4f15d-78ea-4803-9d9d-ad445710d934-istiod-ca-cert\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.446230    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-workload-certs\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.446383    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-credential-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.446526    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.446549    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrfq8\" (UniqueName: \"kubernetes.io/projected/13349545-8422-4466-9152-bd143810b334-kube-api-access-xrfq8\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.446594    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.446608    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.446621    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13349545-8422-4466-9152-bd143810b334-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.447031    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-credential-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.447286    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-workload-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.449381    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc4f15d-78ea-4803-9d9d-ad445710d934-istiod-ca-cert\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.449971    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-data\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.458417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.450268    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-workload-certs\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.482628 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.482582    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cnx4\" (UniqueName: \"kubernetes.io/projected/acc4f15d-78ea-4803-9d9d-ad445710d934-kube-api-access-8cnx4\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.485600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.485572    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-token\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.496250 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.496220    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-podinfo\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.530315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.528999    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-envoy\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.551315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.538715    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13349545-8422-4466-9152-bd143810b334-tls-certs" pod "13349545-8422-4466-9152-bd143810b334" (UID: "13349545-8422-4466-9152-bd143810b334"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:45:11.553053 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.553017    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13349545-8422-4466-9152-bd143810b334-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:11.824111 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.824017    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:11.968220 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.968055    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"
Oct 31 13:45:11.968220 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.968075    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq" event={"ID":"13349545-8422-4466-9152-bd143810b334","Type":"ContainerDied","Data":"763b32e545c2d204628b4c5ed3389402ebe5faaabcdd1c52d6e61a82de5d6f7d"}
Oct 31 13:45:11.968220 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:11.968117    4660 scope.go:122] "RemoveContainer" containerID="f790a4b7f6c455619d4d9d3e7e5ac3770a55a4065698033f25c774766bd5b2e3"
Oct 31 13:45:12.006983 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:12.006274    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"]
Oct 31 13:45:12.017897 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:12.017868    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-fb50fa/test-model-fb50fa-kserve-548bdc8b6c-nj6kq"]
Oct 31 13:45:12.155594 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:12.155560    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"]
Oct 31 13:45:12.164534 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:12.164494    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:45:12.164638 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:12.164583    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:45:12.164676 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:12.164657    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:45:12.536930 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:12.536851    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13349545-8422-4466-9152-bd143810b334" path="/var/lib/kubelet/pods/13349545-8422-4466-9152-bd143810b334/volumes"
Oct 31 13:45:12.984053 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:12.984014    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx" event={"ID":"acc4f15d-78ea-4803-9d9d-ad445710d934","Type":"ContainerStarted","Data":"10cc3753da99e65c253ce224644c71ce661f1539bda8f6970036545aa220ac2a"}
Oct 31 13:45:12.984562 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:12.984076    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx" event={"ID":"acc4f15d-78ea-4803-9d9d-ad445710d934","Type":"ContainerStarted","Data":"8a2eb533b513651c78b378b138b9853ad30d31a832585c9196e88686763930e4"}
Oct 31 13:45:13.019010 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:13.018949    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx" podStartSLOduration=2.01893139 podStartE2EDuration="2.01893139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:13.012563443 +0000 UTC m=+2512.631222608" watchObservedRunningTime="2026-10-31 13:45:13.01893139 +0000 UTC m=+2512.637590556"
Oct 31 13:45:13.825170 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:13.825139    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:13.833872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:13.833846    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:13.996279 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:13.996246    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:13.997333 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:13.997314    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:45:14.289138 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.289104    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"]
Oct 31 13:45:14.290592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.290564    4660 cpu_manager.go:414] "containerMap: removing container" podUID="13349545-8422-4466-9152-bd143810b334" containerName="main"
Oct 31 13:45:14.290728 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.290714    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13349545-8422-4466-9152-bd143810b334" containerName="main"
Oct 31 13:45:14.291088 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.291068    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="13349545-8422-4466-9152-bd143810b334" containerName="main"
Oct 31 13:45:14.316208 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.316169    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"]
Oct 31 13:45:14.316376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.316286    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.331152 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.330770    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1ddade\"/\"test-model-1ddade-kserve-self-signed-certs\""
Oct 31 13:45:14.331152 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.330852    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1ddade\"/\"openshift-service-ca.crt\""
Oct 31 13:45:14.331152 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.330920    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1ddade\"/\"default-dockercfg-vqljc\""
Oct 31 13:45:14.335760 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.333445    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1ddade\"/\"kube-root-ca.crt\""
Oct 31 13:45:14.452950 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.452914    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-tls-certs\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.453180 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.453063    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-home\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.453653 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.453627    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79lzm\" (UniqueName: \"kubernetes.io/projected/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-kube-api-access-79lzm\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.453756 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.453684    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-dshm\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.453804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.453770    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-tmp-dir\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.453874 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.453859    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-model-cache\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.524317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.524116    4660 scope.go:122] "RemoveContainer" containerID="a88b5de348a86e548b6a415a58ac0ad6ea2ff4d992dce31240d9d5164be510aa"
Oct 31 13:45:14.566511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.555536    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-home\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.566511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.555705    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79lzm\" (UniqueName: \"kubernetes.io/projected/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-kube-api-access-79lzm\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.566511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.555755    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-dshm\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.566511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.555838    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-tmp-dir\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.566511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.555941    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-model-cache\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.566511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.556008    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-tls-certs\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.566511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.557795    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-home\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.566511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.558042    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-tmp-dir\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.566511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.558278    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-model-cache\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.612318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.611571    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-dshm\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.621326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.617731    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79lzm\" (UniqueName: \"kubernetes.io/projected/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-kube-api-access-79lzm\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.669318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.651731    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-tls-certs\") pod \"test-model-1ddade-kserve-5c5bcc7bf-5487z\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") " pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:14.689318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:14.671221    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:15.200774 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:15.200741    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"]
Oct 31 13:45:15.520247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:15.520175    4660 scope.go:122] "RemoveContainer" containerID="e372f33e9356f3238d2b8956d85eaab5fec4571662993d2242c3c7d7db0f5837"
Oct 31 13:45:16.041491 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:16.041461    4660 generic.go:356] "Generic (PLEG): container finished" podID="a37b859d-6ef3-4e77-9459-47939791f9c6" containerID="6154ed74364fb9a56286e55338885f3dcb47481dd68b91e118d967932bdcae80" exitCode=60
Oct 31 13:45:16.041632 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:16.041533    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v" event={"ID":"a37b859d-6ef3-4e77-9459-47939791f9c6","Type":"ContainerDied","Data":"6154ed74364fb9a56286e55338885f3dcb47481dd68b91e118d967932bdcae80"}
Oct 31 13:45:16.041632 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:16.041607    4660 scope.go:122] "RemoveContainer" containerID="a88b5de348a86e548b6a415a58ac0ad6ea2ff4d992dce31240d9d5164be510aa"
Oct 31 13:45:16.042230 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:16.042056    4660 scope.go:122] "RemoveContainer" containerID="6154ed74364fb9a56286e55338885f3dcb47481dd68b91e118d967932bdcae80"
Oct 31 13:45:16.042442 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:16.042413    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v_odh-ai-gateway-infra(a37b859d-6ef3-4e77-9459-47939791f9c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v" podUID="a37b859d-6ef3-4e77-9459-47939791f9c6"
Oct 31 13:45:16.044516 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:16.044481    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z" event={"ID":"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9","Type":"ContainerStarted","Data":"c974ad7b6be45d79c033a749587077eace33a22563c1d2c5e1a1c46e04ba30c6"}
Oct 31 13:45:16.044605 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:16.044541    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z" event={"ID":"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9","Type":"ContainerStarted","Data":"8666a0c111be0f53c9c64646acf871225556f235e6876d35c1f6899f869f97ff"}
Oct 31 13:45:16.047654 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:16.047630    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl" event={"ID":"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0","Type":"ContainerStarted","Data":"18bce932d50533888d1b07728b31f5cf88d2b39ecd45d55b1bd33b2357fd64b3"}
Oct 31 13:45:16.107826 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:16.107721    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z" podStartSLOduration=2.107708871 podStartE2EDuration="2.107708871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:16.10245126 +0000 UTC m=+2515.721110426" watchObservedRunningTime="2026-10-31 13:45:16.107708871 +0000 UTC m=+2515.726368035"
Oct 31 13:45:16.521812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:16.521778    4660 scope.go:122] "RemoveContainer" containerID="6139cca17ff0f165e73697af5dedcd570542081b82ec5f7ce6d2a0f7e27ca166"
Oct 31 13:45:16.713725 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:16.713696    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:17.127315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:17.122691    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4" event={"ID":"94082533-524e-4201-ae61-c73877895d65","Type":"ContainerStarted","Data":"d2b6863f8bf706e32bcf063402d4963d18917cea824ae30dab8f5ce41d87115f"}
Oct 31 13:45:17.177653 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:17.177622    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v"]
Oct 31 13:45:17.182796 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:17.182768    4660 generic.go:356] "Generic (PLEG): container finished" podID="bb79fc7d-a4bb-4e66-b320-eeff445ff5c0" containerID="18bce932d50533888d1b07728b31f5cf88d2b39ecd45d55b1bd33b2357fd64b3" exitCode=60
Oct 31 13:45:17.183078 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:17.183056    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl" event={"ID":"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0","Type":"ContainerDied","Data":"18bce932d50533888d1b07728b31f5cf88d2b39ecd45d55b1bd33b2357fd64b3"}
Oct 31 13:45:17.183227 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:17.183213    4660 scope.go:122] "RemoveContainer" containerID="e372f33e9356f3238d2b8956d85eaab5fec4571662993d2242c3c7d7db0f5837"
Oct 31 13:45:17.184326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:17.183834    4660 scope.go:122] "RemoveContainer" containerID="18bce932d50533888d1b07728b31f5cf88d2b39ecd45d55b1bd33b2357fd64b3"
Oct 31 13:45:17.184947 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:17.184919    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl_odh-ai-gateway-infra(bb79fc7d-a4bb-4e66-b320-eeff445ff5c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl" podUID="bb79fc7d-a4bb-4e66-b320-eeff445ff5c0"
Oct 31 13:45:17.207452 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:17.207349    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl"]
Oct 31 13:45:17.455561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:17.455531    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v"
Oct 31 13:45:17.636255 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:17.636227    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37b859d-6ef3-4e77-9459-47939791f9c6-kube-api-access-kb4vn\" (UniqueName: \"kubernetes.io/projected/a37b859d-6ef3-4e77-9459-47939791f9c6-kube-api-access-kb4vn\") pod \"a37b859d-6ef3-4e77-9459-47939791f9c6\" (UID: \"a37b859d-6ef3-4e77-9459-47939791f9c6\") "
Oct 31 13:45:17.636589 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:17.636572    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a37b859d-6ef3-4e77-9459-47939791f9c6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37b859d-6ef3-4e77-9459-47939791f9c6-maas-api-service-ca\") pod \"a37b859d-6ef3-4e77-9459-47939791f9c6\" (UID: \"a37b859d-6ef3-4e77-9459-47939791f9c6\") "
Oct 31 13:45:17.637007 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:17.636976    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37b859d-6ef3-4e77-9459-47939791f9c6-maas-api-service-ca" pod "a37b859d-6ef3-4e77-9459-47939791f9c6" (UID: "a37b859d-6ef3-4e77-9459-47939791f9c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:17.637620 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:17.637597    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37b859d-6ef3-4e77-9459-47939791f9c6-maas-api-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:17.638568 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:17.638540    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37b859d-6ef3-4e77-9459-47939791f9c6-kube-api-access-kb4vn" pod "a37b859d-6ef3-4e77-9459-47939791f9c6" (UID: "a37b859d-6ef3-4e77-9459-47939791f9c6"). InnerVolumeSpecName "kube-api-access-kb4vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:17.739532 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:17.739499    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb4vn\" (UniqueName: \"kubernetes.io/projected/a37b859d-6ef3-4e77-9459-47939791f9c6-kube-api-access-kb4vn\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:18.207269 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.207237    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v"
Oct 31 13:45:18.207464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.207262    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v" event={"ID":"a37b859d-6ef3-4e77-9459-47939791f9c6","Type":"ContainerDied","Data":"38727c0981767885ea4013d6f007f2f6a713f24d23403f3970565cbb2120bde7"}
Oct 31 13:45:18.207464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.207385    4660 scope.go:122] "RemoveContainer" containerID="6154ed74364fb9a56286e55338885f3dcb47481dd68b91e118d967932bdcae80"
Oct 31 13:45:18.212249 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.212219    4660 generic.go:356] "Generic (PLEG): container finished" podID="94082533-524e-4201-ae61-c73877895d65" containerID="d2b6863f8bf706e32bcf063402d4963d18917cea824ae30dab8f5ce41d87115f" exitCode=60
Oct 31 13:45:18.212361 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.212260    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4" event={"ID":"94082533-524e-4201-ae61-c73877895d65","Type":"ContainerDied","Data":"d2b6863f8bf706e32bcf063402d4963d18917cea824ae30dab8f5ce41d87115f"}
Oct 31 13:45:18.213052 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.213029    4660 scope.go:122] "RemoveContainer" containerID="d2b6863f8bf706e32bcf063402d4963d18917cea824ae30dab8f5ce41d87115f"
Oct 31 13:45:18.213490 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:18.213463    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4_odh-ai-gateway-infra(94082533-524e-4201-ae61-c73877895d65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4" podUID="94082533-524e-4201-ae61-c73877895d65"
Oct 31 13:45:18.227506 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.227474    4660 scope.go:122] "RemoveContainer" containerID="6139cca17ff0f165e73697af5dedcd570542081b82ec5f7ce6d2a0f7e27ca166"
Oct 31 13:45:18.237640 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.237612    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4"]
Oct 31 13:45:18.352362 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.349251    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v"]
Oct 31 13:45:18.355898 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.355870    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890905-vtw5v"]
Oct 31 13:45:18.425355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.421564    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl"
Oct 31 13:45:18.520110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.520023    4660 scope.go:122] "RemoveContainer" containerID="b7ff63489c0205eb6359942724975fb631e5190626a485e6ced122df1cee73fd"
Oct 31 13:45:18.520278 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.520163    4660 scope.go:122] "RemoveContainer" containerID="54c7fb71e8ab2aa9d787f1bd1f8db6b13ffd44f5de83c4a2153a11078319dea3"
Oct 31 13:45:18.521401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.520983    4660 scope.go:122] "RemoveContainer" containerID="4d182d2f32fdf3ffb619062eead50b88aa92132ba5659472493952aedaf749a8"
Oct 31 13:45:18.533761 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.533738    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a37b859d-6ef3-4e77-9459-47939791f9c6" path="/var/lib/kubelet/pods/a37b859d-6ef3-4e77-9459-47939791f9c6/volumes"
Oct 31 13:45:18.557202 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.557176    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb79fc7d-a4bb-4e66-b320-eeff445ff5c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb79fc7d-a4bb-4e66-b320-eeff445ff5c0-maas-api-service-ca\") pod \"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0\" (UID: \"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0\") "
Oct 31 13:45:18.557345 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.557320    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb79fc7d-a4bb-4e66-b320-eeff445ff5c0-kube-api-access-ftwfw\" (UniqueName: \"kubernetes.io/projected/bb79fc7d-a4bb-4e66-b320-eeff445ff5c0-kube-api-access-ftwfw\") pod \"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0\" (UID: \"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0\") "
Oct 31 13:45:18.557553 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.557527    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb79fc7d-a4bb-4e66-b320-eeff445ff5c0-maas-api-service-ca" pod "bb79fc7d-a4bb-4e66-b320-eeff445ff5c0" (UID: "bb79fc7d-a4bb-4e66-b320-eeff445ff5c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:18.557917 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.557899    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb79fc7d-a4bb-4e66-b320-eeff445ff5c0-maas-api-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:18.559579 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.559552    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb79fc7d-a4bb-4e66-b320-eeff445ff5c0-kube-api-access-ftwfw" pod "bb79fc7d-a4bb-4e66-b320-eeff445ff5c0" (UID: "bb79fc7d-a4bb-4e66-b320-eeff445ff5c0"). InnerVolumeSpecName "kube-api-access-ftwfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:18.659518 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:18.659488    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftwfw\" (UniqueName: \"kubernetes.io/projected/bb79fc7d-a4bb-4e66-b320-eeff445ff5c0-kube-api-access-ftwfw\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:19.224623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:19.224588    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl" event={"ID":"bb79fc7d-a4bb-4e66-b320-eeff445ff5c0","Type":"ContainerDied","Data":"3e88181c2d9a14af0b5f01cc22530b89f2c4a01b77adcf8aca94d05903ca5b9b"}
Oct 31 13:45:19.224780 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:19.224665    4660 scope.go:122] "RemoveContainer" containerID="18bce932d50533888d1b07728b31f5cf88d2b39ecd45d55b1bd33b2357fd64b3"
Oct 31 13:45:19.224780 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:19.224595    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl"
Oct 31 13:45:19.228928 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:19.228902    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m" event={"ID":"afa08b46-4029-44a9-9c58-1ae42760deb7","Type":"ContainerStarted","Data":"f1ae724b353d4da17d54d723cd09e42ca8703399b36d0d23fcd482afdc4dc1da"}
Oct 31 13:45:19.231128 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:19.231106    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld" event={"ID":"61588bc9-c966-4ea3-9a38-750555e43840","Type":"ContainerStarted","Data":"f521ab6a8a1cd3a50bcf30d7c9c65821ad2cd3cf7b761200360daadb5030f6a0"}
Oct 31 13:45:19.233557 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:19.233530    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7" event={"ID":"ad8e87df-0d34-443a-ba4a-11f647aed571","Type":"ContainerStarted","Data":"ac19a165fe166d0f0c8ef9986fc451a976b22c2c82123b5c0c0fabafd53aa0d2"}
Oct 31 13:45:19.290557 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:19.290526    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl"]
Oct 31 13:45:19.313352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:19.307471    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-554759-29890905-9dvzl"]
Oct 31 13:45:19.475380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:19.475318    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4"
Oct 31 13:45:19.582845 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:19.582815    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94082533-524e-4201-ae61-c73877895d65-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94082533-524e-4201-ae61-c73877895d65-maas-api-service-ca\") pod \"94082533-524e-4201-ae61-c73877895d65\" (UID: \"94082533-524e-4201-ae61-c73877895d65\") "
Oct 31 13:45:19.583207 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:19.583107    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94082533-524e-4201-ae61-c73877895d65-kube-api-access-wcf7j\" (UniqueName: \"kubernetes.io/projected/94082533-524e-4201-ae61-c73877895d65-kube-api-access-wcf7j\") pod \"94082533-524e-4201-ae61-c73877895d65\" (UID: \"94082533-524e-4201-ae61-c73877895d65\") "
Oct 31 13:45:19.583646 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:19.583619    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94082533-524e-4201-ae61-c73877895d65-maas-api-service-ca" pod "94082533-524e-4201-ae61-c73877895d65" (UID: "94082533-524e-4201-ae61-c73877895d65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:19.583945 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:19.583924    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94082533-524e-4201-ae61-c73877895d65-maas-api-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:19.604285 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:19.601832    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94082533-524e-4201-ae61-c73877895d65-kube-api-access-wcf7j" pod "94082533-524e-4201-ae61-c73877895d65" (UID: "94082533-524e-4201-ae61-c73877895d65"). InnerVolumeSpecName "kube-api-access-wcf7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:19.685186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:19.685152    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcf7j\" (UniqueName: \"kubernetes.io/projected/94082533-524e-4201-ae61-c73877895d65-kube-api-access-wcf7j\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:20.259642 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.259603    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4" event={"ID":"94082533-524e-4201-ae61-c73877895d65","Type":"ContainerDied","Data":"6201b3a71f0228f0971f1dd3bfe6d75ba4d64162010f23c82ec0eda419d8afe8"}
Oct 31 13:45:20.259883 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.259860    4660 scope.go:122] "RemoveContainer" containerID="d2b6863f8bf706e32bcf063402d4963d18917cea824ae30dab8f5ce41d87115f"
Oct 31 13:45:20.260287 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.260252    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4"
Oct 31 13:45:20.267372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.267340    4660 generic.go:356] "Generic (PLEG): container finished" podID="afa08b46-4029-44a9-9c58-1ae42760deb7" containerID="f1ae724b353d4da17d54d723cd09e42ca8703399b36d0d23fcd482afdc4dc1da" exitCode=60
Oct 31 13:45:20.267548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.267499    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m" event={"ID":"afa08b46-4029-44a9-9c58-1ae42760deb7","Type":"ContainerDied","Data":"f1ae724b353d4da17d54d723cd09e42ca8703399b36d0d23fcd482afdc4dc1da"}
Oct 31 13:45:20.267921 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.267904    4660 scope.go:122] "RemoveContainer" containerID="f1ae724b353d4da17d54d723cd09e42ca8703399b36d0d23fcd482afdc4dc1da"
Oct 31 13:45:20.268261 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:20.268242    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m_odh-ai-gateway-infra(afa08b46-4029-44a9-9c58-1ae42760deb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m" podUID="afa08b46-4029-44a9-9c58-1ae42760deb7"
Oct 31 13:45:20.282035 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.282005    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m"]
Oct 31 13:45:20.284623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.284599    4660 generic.go:356] "Generic (PLEG): container finished" podID="61588bc9-c966-4ea3-9a38-750555e43840" containerID="f521ab6a8a1cd3a50bcf30d7c9c65821ad2cd3cf7b761200360daadb5030f6a0" exitCode=60
Oct 31 13:45:20.284748 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.284684    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld" event={"ID":"61588bc9-c966-4ea3-9a38-750555e43840","Type":"ContainerDied","Data":"f521ab6a8a1cd3a50bcf30d7c9c65821ad2cd3cf7b761200360daadb5030f6a0"}
Oct 31 13:45:20.285259 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.285229    4660 scope.go:122] "RemoveContainer" containerID="f521ab6a8a1cd3a50bcf30d7c9c65821ad2cd3cf7b761200360daadb5030f6a0"
Oct 31 13:45:20.285745 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:20.285707    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld_odh-ai-gateway-infra(61588bc9-c966-4ea3-9a38-750555e43840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld" podUID="61588bc9-c966-4ea3-9a38-750555e43840"
Oct 31 13:45:20.285939 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.285896    4660 scope.go:122] "RemoveContainer" containerID="b7ff63489c0205eb6359942724975fb631e5190626a485e6ced122df1cee73fd"
Oct 31 13:45:20.287817 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.287794    4660 generic.go:356] "Generic (PLEG): container finished" podID="ad8e87df-0d34-443a-ba4a-11f647aed571" containerID="ac19a165fe166d0f0c8ef9986fc451a976b22c2c82123b5c0c0fabafd53aa0d2" exitCode=60
Oct 31 13:45:20.287908 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.287864    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7" event={"ID":"ad8e87df-0d34-443a-ba4a-11f647aed571","Type":"ContainerDied","Data":"ac19a165fe166d0f0c8ef9986fc451a976b22c2c82123b5c0c0fabafd53aa0d2"}
Oct 31 13:45:20.288223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.288202    4660 scope.go:122] "RemoveContainer" containerID="ac19a165fe166d0f0c8ef9986fc451a976b22c2c82123b5c0c0fabafd53aa0d2"
Oct 31 13:45:20.288574 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:20.288554    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7_odh-ai-gateway-infra(ad8e87df-0d34-443a-ba4a-11f647aed571)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7" podUID="ad8e87df-0d34-443a-ba4a-11f647aed571"
Oct 31 13:45:20.303770 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.303747    4660 scope.go:122] "RemoveContainer" containerID="54c7fb71e8ab2aa9d787f1bd1f8db6b13ffd44f5de83c4a2153a11078319dea3"
Oct 31 13:45:20.318352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.318322    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4"]
Oct 31 13:45:20.337751 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.337565    4660 scope.go:122] "RemoveContainer" containerID="4d182d2f32fdf3ffb619062eead50b88aa92132ba5659472493952aedaf749a8"
Oct 31 13:45:20.357035 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.357005    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890905-j46l4"]
Oct 31 13:45:20.365191 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.365164    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld"]
Oct 31 13:45:20.376201 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.376168    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7"]
Oct 31 13:45:20.533252 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.533173    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94082533-524e-4201-ae61-c73877895d65" path="/var/lib/kubelet/pods/94082533-524e-4201-ae61-c73877895d65/volumes"
Oct 31 13:45:20.533905 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:20.533885    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb79fc7d-a4bb-4e66-b320-eeff445ff5c0" path="/var/lib/kubelet/pods/bb79fc7d-a4bb-4e66-b320-eeff445ff5c0/volumes"
Oct 31 13:45:20.748647 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:20.748606    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:45:21.174845 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:21.174809    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"]
Oct 31 13:45:22.059495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.048538    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld"
Oct 31 13:45:22.072572 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.072508    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7"
Oct 31 13:45:22.110736 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.110707    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m"
Oct 31 13:45:22.203842 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.203806    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61588bc9-c966-4ea3-9a38-750555e43840-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61588bc9-c966-4ea3-9a38-750555e43840-maas-api-service-ca\") pod \"61588bc9-c966-4ea3-9a38-750555e43840\" (UID: \"61588bc9-c966-4ea3-9a38-750555e43840\") "
Oct 31 13:45:22.204124 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.204104    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad8e87df-0d34-443a-ba4a-11f647aed571-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8e87df-0d34-443a-ba4a-11f647aed571-maas-api-service-ca\") pod \"ad8e87df-0d34-443a-ba4a-11f647aed571\" (UID: \"ad8e87df-0d34-443a-ba4a-11f647aed571\") "
Oct 31 13:45:22.204274 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.204162    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61588bc9-c966-4ea3-9a38-750555e43840-kube-api-access-8kk2x\" (UniqueName: \"kubernetes.io/projected/61588bc9-c966-4ea3-9a38-750555e43840-kube-api-access-8kk2x\") pod \"61588bc9-c966-4ea3-9a38-750555e43840\" (UID: \"61588bc9-c966-4ea3-9a38-750555e43840\") "
Oct 31 13:45:22.204274 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.204200    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad8e87df-0d34-443a-ba4a-11f647aed571-kube-api-access-6xmvq\" (UniqueName: \"kubernetes.io/projected/ad8e87df-0d34-443a-ba4a-11f647aed571-kube-api-access-6xmvq\") pod \"ad8e87df-0d34-443a-ba4a-11f647aed571\" (UID: \"ad8e87df-0d34-443a-ba4a-11f647aed571\") "
Oct 31 13:45:22.204274 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.204245    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afa08b46-4029-44a9-9c58-1ae42760deb7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa08b46-4029-44a9-9c58-1ae42760deb7-maas-api-service-ca\") pod \"afa08b46-4029-44a9-9c58-1ae42760deb7\" (UID: \"afa08b46-4029-44a9-9c58-1ae42760deb7\") "
Oct 31 13:45:22.204631 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.204608    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8e87df-0d34-443a-ba4a-11f647aed571-maas-api-service-ca" pod "ad8e87df-0d34-443a-ba4a-11f647aed571" (UID: "ad8e87df-0d34-443a-ba4a-11f647aed571"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:22.204734 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.204647    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa08b46-4029-44a9-9c58-1ae42760deb7-maas-api-service-ca" pod "afa08b46-4029-44a9-9c58-1ae42760deb7" (UID: "afa08b46-4029-44a9-9c58-1ae42760deb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:22.204734 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.204673    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61588bc9-c966-4ea3-9a38-750555e43840-maas-api-service-ca" pod "61588bc9-c966-4ea3-9a38-750555e43840" (UID: "61588bc9-c966-4ea3-9a38-750555e43840"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:22.205110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.205087    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8e87df-0d34-443a-ba4a-11f647aed571-maas-api-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:22.205189 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.205117    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa08b46-4029-44a9-9c58-1ae42760deb7-maas-api-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:22.205189 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.205135    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61588bc9-c966-4ea3-9a38-750555e43840-maas-api-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:22.208003 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.206996    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61588bc9-c966-4ea3-9a38-750555e43840-kube-api-access-8kk2x" pod "61588bc9-c966-4ea3-9a38-750555e43840" (UID: "61588bc9-c966-4ea3-9a38-750555e43840"). InnerVolumeSpecName "kube-api-access-8kk2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:22.208090 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.208004    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8e87df-0d34-443a-ba4a-11f647aed571-kube-api-access-6xmvq" pod "ad8e87df-0d34-443a-ba4a-11f647aed571" (UID: "ad8e87df-0d34-443a-ba4a-11f647aed571"). InnerVolumeSpecName "kube-api-access-6xmvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:22.309351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.309231    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa08b46-4029-44a9-9c58-1ae42760deb7-kube-api-access-lvhs7\" (UniqueName: \"kubernetes.io/projected/afa08b46-4029-44a9-9c58-1ae42760deb7-kube-api-access-lvhs7\") pod \"afa08b46-4029-44a9-9c58-1ae42760deb7\" (UID: \"afa08b46-4029-44a9-9c58-1ae42760deb7\") "
Oct 31 13:45:22.310153 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.310070    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kk2x\" (UniqueName: \"kubernetes.io/projected/61588bc9-c966-4ea3-9a38-750555e43840-kube-api-access-8kk2x\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:22.310153 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.310101    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xmvq\" (UniqueName: \"kubernetes.io/projected/ad8e87df-0d34-443a-ba4a-11f647aed571-kube-api-access-6xmvq\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:22.325073 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.325036    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa08b46-4029-44a9-9c58-1ae42760deb7-kube-api-access-lvhs7" pod "afa08b46-4029-44a9-9c58-1ae42760deb7" (UID: "afa08b46-4029-44a9-9c58-1ae42760deb7"). InnerVolumeSpecName "kube-api-access-lvhs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:22.411111 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.411078    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvhs7\" (UniqueName: \"kubernetes.io/projected/afa08b46-4029-44a9-9c58-1ae42760deb7-kube-api-access-lvhs7\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:22.479096 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.479055    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m"
Oct 31 13:45:22.479286 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.479136    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m" event={"ID":"afa08b46-4029-44a9-9c58-1ae42760deb7","Type":"ContainerDied","Data":"3406c983f14071b3e47dbf644826c2e498770af1989af2d98cb42253de77875f"}
Oct 31 13:45:22.479286 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.479190    4660 scope.go:122] "RemoveContainer" containerID="f1ae724b353d4da17d54d723cd09e42ca8703399b36d0d23fcd482afdc4dc1da"
Oct 31 13:45:22.497339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.485125    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld"
Oct 31 13:45:22.497339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.487042    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld" event={"ID":"61588bc9-c966-4ea3-9a38-750555e43840","Type":"ContainerDied","Data":"a3caa2e31591024dd2bf42711d83a31be3c56534714b840d5a34cc2875304bef"}
Oct 31 13:45:22.497339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.491500    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7"
Oct 31 13:45:22.497339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.491618    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7" event={"ID":"ad8e87df-0d34-443a-ba4a-11f647aed571","Type":"ContainerDied","Data":"78b701eb72c5e75a7f5d4d27ad89146bd3a59e2b916ab15a3d8c93de3c6eaffa"}
Oct 31 13:45:22.515110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.515081    4660 scope.go:122] "RemoveContainer" containerID="f521ab6a8a1cd3a50bcf30d7c9c65821ad2cd3cf7b761200360daadb5030f6a0"
Oct 31 13:45:22.556363 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.556334    4660 scope.go:122] "RemoveContainer" containerID="ac19a165fe166d0f0c8ef9986fc451a976b22c2c82123b5c0c0fabafd53aa0d2"
Oct 31 13:45:22.561786 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.561727    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld"]
Oct 31 13:45:22.571338 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.570368    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890905-bc7ld"]
Oct 31 13:45:22.573523 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.573499    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7"]
Oct 31 13:45:22.578344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.578318    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890905-2sff7"]
Oct 31 13:45:22.597977 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.597952    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m"]
Oct 31 13:45:22.605410 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:22.605381    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-35b584-29890905-9w72m"]
Oct 31 13:45:24.538718 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:24.538684    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61588bc9-c966-4ea3-9a38-750555e43840" path="/var/lib/kubelet/pods/61588bc9-c966-4ea3-9a38-750555e43840/volumes"
Oct 31 13:45:24.539375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:24.539355    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad8e87df-0d34-443a-ba4a-11f647aed571" path="/var/lib/kubelet/pods/ad8e87df-0d34-443a-ba4a-11f647aed571/volumes"
Oct 31 13:45:24.539916 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:24.539899    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa08b46-4029-44a9-9c58-1ae42760deb7" path="/var/lib/kubelet/pods/afa08b46-4029-44a9-9c58-1ae42760deb7/volumes"
Oct 31 13:45:24.713857 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:24.713821    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:24.734713 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:24.734626    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:25.541894 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:25.541862    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:45:31.001009 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:30.996722    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"]
Oct 31 13:45:31.001009 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:30.997442    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" containerID="cri-o://c4d7f140762fc842a63c223b2af56c155d1157c76436a62771308a265bc1288f" gracePeriod=30
Oct 31 13:45:31.498088 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:31.498051    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:45:31.498283 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:31.498115    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:45:36.236369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.236323    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz" podUID="f3e60e1e-371d-4a37-b988-e76510d17e90" containerName="main" containerID="cri-o://aff6c5fd25fb611d6782659427544f9f25c751a6c5d4e3ef8f4df514b3b5806b" gracePeriod=45
Oct 31 13:45:36.384472 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.384439    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"]
Oct 31 13:45:36.400846 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.400802    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.483358    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-istio-data\") pod \"7b0354dc-4369-44e8-a995-5018598ea92a\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") "
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.483426    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0354dc-4369-44e8-a995-5018598ea92a-istio-token\" (UniqueName: \"kubernetes.io/projected/7b0354dc-4369-44e8-a995-5018598ea92a-istio-token\") pod \"7b0354dc-4369-44e8-a995-5018598ea92a\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") "
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.483485    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-workload-socket\") pod \"7b0354dc-4369-44e8-a995-5018598ea92a\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") "
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.483634    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-istio-envoy\") pod \"7b0354dc-4369-44e8-a995-5018598ea92a\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") "
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.483758    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0354dc-4369-44e8-a995-5018598ea92a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b0354dc-4369-44e8-a995-5018598ea92a-istiod-ca-cert\") pod \"7b0354dc-4369-44e8-a995-5018598ea92a\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") "
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.483800    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-workload-certs\") pod \"7b0354dc-4369-44e8-a995-5018598ea92a\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") "
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.483792    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-istio-data" pod "7b0354dc-4369-44e8-a995-5018598ea92a" (UID: "7b0354dc-4369-44e8-a995-5018598ea92a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.483863    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b0354dc-4369-44e8-a995-5018598ea92a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b0354dc-4369-44e8-a995-5018598ea92a-istio-podinfo\") pod \"7b0354dc-4369-44e8-a995-5018598ea92a\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") "
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.483913    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-credential-socket\") pod \"7b0354dc-4369-44e8-a995-5018598ea92a\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") "
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.484976    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-workload-certs" pod "7b0354dc-4369-44e8-a995-5018598ea92a" (UID: "7b0354dc-4369-44e8-a995-5018598ea92a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.485246    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-credential-socket" pod "7b0354dc-4369-44e8-a995-5018598ea92a" (UID: "7b0354dc-4369-44e8-a995-5018598ea92a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.485514    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0354dc-4369-44e8-a995-5018598ea92a-istiod-ca-cert" pod "7b0354dc-4369-44e8-a995-5018598ea92a" (UID: "7b0354dc-4369-44e8-a995-5018598ea92a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.485570    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-workload-socket" pod "7b0354dc-4369-44e8-a995-5018598ea92a" (UID: "7b0354dc-4369-44e8-a995-5018598ea92a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.487373    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0354dc-4369-44e8-a995-5018598ea92a-kube-api-access-rf6js\" (UniqueName: \"kubernetes.io/projected/7b0354dc-4369-44e8-a995-5018598ea92a-kube-api-access-rf6js\") pod \"7b0354dc-4369-44e8-a995-5018598ea92a\" (UID: \"7b0354dc-4369-44e8-a995-5018598ea92a\") "
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.487478    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-istio-envoy" pod "7b0354dc-4369-44e8-a995-5018598ea92a" (UID: "7b0354dc-4369-44e8-a995-5018598ea92a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.488317    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b0354dc-4369-44e8-a995-5018598ea92a-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.488341    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.488355    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.488370    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.488395    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.488408    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b0354dc-4369-44e8-a995-5018598ea92a-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:36.489471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.489371    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b0354dc-4369-44e8-a995-5018598ea92a-istio-podinfo" pod "7b0354dc-4369-44e8-a995-5018598ea92a" (UID: "7b0354dc-4369-44e8-a995-5018598ea92a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:45:36.490888 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.490864    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0354dc-4369-44e8-a995-5018598ea92a-kube-api-access-rf6js" pod "7b0354dc-4369-44e8-a995-5018598ea92a" (UID: "7b0354dc-4369-44e8-a995-5018598ea92a"). InnerVolumeSpecName "kube-api-access-rf6js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:36.492425 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.492390    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0354dc-4369-44e8-a995-5018598ea92a-istio-token" pod "7b0354dc-4369-44e8-a995-5018598ea92a" (UID: "7b0354dc-4369-44e8-a995-5018598ea92a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:36.619389 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.608906    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b0354dc-4369-44e8-a995-5018598ea92a-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:36.619389 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.608952    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf6js\" (UniqueName: \"kubernetes.io/projected/7b0354dc-4369-44e8-a995-5018598ea92a-kube-api-access-rf6js\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:36.619389 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.608970    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b0354dc-4369-44e8-a995-5018598ea92a-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:36.798593 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.798559    4660 generic.go:356] "Generic (PLEG): container finished" podID="7b0354dc-4369-44e8-a995-5018598ea92a" containerID="c4d7f140762fc842a63c223b2af56c155d1157c76436a62771308a265bc1288f" exitCode=0
Oct 31 13:45:36.799169 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.799145    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" event={"ID":"7b0354dc-4369-44e8-a995-5018598ea92a","Type":"ContainerDied","Data":"c4d7f140762fc842a63c223b2af56c155d1157c76436a62771308a265bc1288f"}
Oct 31 13:45:36.799372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.799357    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9" event={"ID":"7b0354dc-4369-44e8-a995-5018598ea92a","Type":"ContainerDied","Data":"41bb2b1fb56bf50b918bc74b37f6ce090f628ad42804e89867a3489a40afab3b"}
Oct 31 13:45:36.799638 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.799608    4660 scope.go:122] "RemoveContainer" containerID="c4d7f140762fc842a63c223b2af56c155d1157c76436a62771308a265bc1288f"
Oct 31 13:45:36.801184 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.801168    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"
Oct 31 13:45:36.815661 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.815490    4660 generic.go:356] "Generic (PLEG): container finished" podID="f3e60e1e-371d-4a37-b988-e76510d17e90" containerID="aff6c5fd25fb611d6782659427544f9f25c751a6c5d4e3ef8f4df514b3b5806b" exitCode=0
Oct 31 13:45:36.815661 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.815599    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz" event={"ID":"f3e60e1e-371d-4a37-b988-e76510d17e90","Type":"ContainerDied","Data":"aff6c5fd25fb611d6782659427544f9f25c751a6c5d4e3ef8f4df514b3b5806b"}
Oct 31 13:45:36.832045 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.832016    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"]
Oct 31 13:45:36.852894 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.852844    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4hwd9"]
Oct 31 13:45:36.878808 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.878783    4660 scope.go:122] "RemoveContainer" containerID="c4d7f140762fc842a63c223b2af56c155d1157c76436a62771308a265bc1288f"
Oct 31 13:45:36.884171 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:36.882436    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d7f140762fc842a63c223b2af56c155d1157c76436a62771308a265bc1288f\": container with ID starting with c4d7f140762fc842a63c223b2af56c155d1157c76436a62771308a265bc1288f not found: ID does not exist" containerID="c4d7f140762fc842a63c223b2af56c155d1157c76436a62771308a265bc1288f"
Oct 31 13:45:36.884171 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.882468    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d7f140762fc842a63c223b2af56c155d1157c76436a62771308a265bc1288f"} err="failed to get container status \"c4d7f140762fc842a63c223b2af56c155d1157c76436a62771308a265bc1288f\": rpc error: code = NotFound desc = could not find container \"c4d7f140762fc842a63c223b2af56c155d1157c76436a62771308a265bc1288f\": container with ID starting with c4d7f140762fc842a63c223b2af56c155d1157c76436a62771308a265bc1288f not found: ID does not exist"
Oct 31 13:45:36.906023 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.905999    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:45:36.948103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.937558    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e60e1e-371d-4a37-b988-e76510d17e90-kube-api-access-tb9vs\" (UniqueName: \"kubernetes.io/projected/f3e60e1e-371d-4a37-b988-e76510d17e90-kube-api-access-tb9vs\") pod \"f3e60e1e-371d-4a37-b988-e76510d17e90\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") "
Oct 31 13:45:36.948103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.937607    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-model-cache\") pod \"f3e60e1e-371d-4a37-b988-e76510d17e90\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") "
Oct 31 13:45:36.948103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.937800    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-dshm\") pod \"f3e60e1e-371d-4a37-b988-e76510d17e90\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") "
Oct 31 13:45:36.948103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.937888    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3e60e1e-371d-4a37-b988-e76510d17e90-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3e60e1e-371d-4a37-b988-e76510d17e90-tls-certs\") pod \"f3e60e1e-371d-4a37-b988-e76510d17e90\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") "
Oct 31 13:45:36.948103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.937985    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-home\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-home\") pod \"f3e60e1e-371d-4a37-b988-e76510d17e90\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") "
Oct 31 13:45:36.948103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.938016    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-model-cache" pod "f3e60e1e-371d-4a37-b988-e76510d17e90" (UID: "f3e60e1e-371d-4a37-b988-e76510d17e90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:36.948103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.938024    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-tmp-dir\") pod \"f3e60e1e-371d-4a37-b988-e76510d17e90\" (UID: \"f3e60e1e-371d-4a37-b988-e76510d17e90\") "
Oct 31 13:45:36.948103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.938239    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-tmp-dir" pod "f3e60e1e-371d-4a37-b988-e76510d17e90" (UID: "f3e60e1e-371d-4a37-b988-e76510d17e90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:36.948103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.938738    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-home" pod "f3e60e1e-371d-4a37-b988-e76510d17e90" (UID: "f3e60e1e-371d-4a37-b988-e76510d17e90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:36.948103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.939246    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:36.948103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.939273    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:36.948103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.939310    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:36.948103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.940277    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e60e1e-371d-4a37-b988-e76510d17e90-kube-api-access-tb9vs" pod "f3e60e1e-371d-4a37-b988-e76510d17e90" (UID: "f3e60e1e-371d-4a37-b988-e76510d17e90"). InnerVolumeSpecName "kube-api-access-tb9vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:36.976313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.957827    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3e60e1e-371d-4a37-b988-e76510d17e90-tls-certs" pod "f3e60e1e-371d-4a37-b988-e76510d17e90" (UID: "f3e60e1e-371d-4a37-b988-e76510d17e90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:45:36.999093 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:36.999043    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-dshm" pod "f3e60e1e-371d-4a37-b988-e76510d17e90" (UID: "f3e60e1e-371d-4a37-b988-e76510d17e90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:37.052079 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:37.051658    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb9vs\" (UniqueName: \"kubernetes.io/projected/f3e60e1e-371d-4a37-b988-e76510d17e90-kube-api-access-tb9vs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:37.052079 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:37.051692    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3e60e1e-371d-4a37-b988-e76510d17e90-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:37.052079 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:37.051716    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3e60e1e-371d-4a37-b988-e76510d17e90-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:37.826487 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:37.826443    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"
Oct 31 13:45:37.827060 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:37.826616    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz" event={"ID":"f3e60e1e-371d-4a37-b988-e76510d17e90","Type":"ContainerDied","Data":"86fe6f00b456b570a042017db6c69c878dbc5d584f92728a05de86331fe7f0ab"}
Oct 31 13:45:37.827060 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:37.826857    4660 scope.go:122] "RemoveContainer" containerID="aff6c5fd25fb611d6782659427544f9f25c751a6c5d4e3ef8f4df514b3b5806b"
Oct 31 13:45:37.857706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:37.857672    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"]
Oct 31 13:45:37.862453 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:37.862432    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-61f295/test-model-61f295-kserve-798c45b56f-6sgqz"]
Oct 31 13:45:38.533279 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:38.533243    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" path="/var/lib/kubelet/pods/7b0354dc-4369-44e8-a995-5018598ea92a/volumes"
Oct 31 13:45:38.534028 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:38.534004    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e60e1e-371d-4a37-b988-e76510d17e90" path="/var/lib/kubelet/pods/f3e60e1e-371d-4a37-b988-e76510d17e90/volumes"
Oct 31 13:45:45.054212 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:45.054159    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"]
Oct 31 13:45:45.054734 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:45.054440    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b" podUID="df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" containerName="istio-proxy" containerID="cri-o://a407c755e738f29adca841b5c8a767ec8f9b6a43416d8da4c875f0bdacc6d36a" gracePeriod=30
Oct 31 13:45:48.160869 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:48.158006    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"]
Oct 31 13:45:50.244734 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.244699    4660 generic.go:356] "Generic (PLEG): container finished" podID="df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" containerID="a407c755e738f29adca841b5c8a767ec8f9b6a43416d8da4c875f0bdacc6d36a" exitCode=0
Oct 31 13:45:50.245282 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.244945    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b" event={"ID":"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097","Type":"ContainerDied","Data":"a407c755e738f29adca841b5c8a767ec8f9b6a43416d8da4c875f0bdacc6d36a"}
Oct 31 13:45:50.437289 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.437258    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:45:50.486065 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.486028    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-workload-certs\") pod \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") "
Oct 31 13:45:50.486254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.486152    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-data\") pod \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") "
Oct 31 13:45:50.486254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.486185    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-token\" (UniqueName: \"kubernetes.io/projected/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-token\") pod \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") "
Oct 31 13:45:50.486254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.486209    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-credential-socket\") pod \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") "
Oct 31 13:45:50.486626 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.486309    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-workload-certs" pod "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" (UID: "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:50.486626 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.486502    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istiod-ca-cert\") pod \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") "
Oct 31 13:45:50.486626 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.486521    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-credential-socket" pod "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" (UID: "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:50.486808 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.486641    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-kube-api-access-fx57f\" (UniqueName: \"kubernetes.io/projected/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-kube-api-access-fx57f\") pod \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") "
Oct 31 13:45:50.486808 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.486685    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-envoy\") pod \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") "
Oct 31 13:45:50.486915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.486859    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istiod-ca-cert" pod "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" (UID: "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:50.486915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.486877    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-workload-socket\") pod \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") "
Oct 31 13:45:50.487025 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.486935    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-podinfo\") pod \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\" (UID: \"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097\") "
Oct 31 13:45:50.487183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.487160    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-data" pod "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" (UID: "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:50.487320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.487281    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-workload-socket" pod "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" (UID: "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:50.487890 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.487862    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.487987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.487896    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.487987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.487915    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.487987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.487931    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.487987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.487948    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.489365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.489280    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-token" pod "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" (UID: "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:50.489548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.489527    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-envoy" pod "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" (UID: "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:50.489629 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.489562    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-kube-api-access-fx57f" pod "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" (UID: "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097"). InnerVolumeSpecName "kube-api-access-fx57f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:50.490676 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.490626    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-podinfo" pod "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" (UID: "df482cd7-2dd2-4cb0-b791-e4a9f8c1e097"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:45:50.589119 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.589083    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.589119 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.589120    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.589381 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.589136    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx57f\" (UniqueName: \"kubernetes.io/projected/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-kube-api-access-fx57f\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.589381 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:50.589153    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.740906 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:50.740813    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:45:51.256958 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:51.256925    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"
Oct 31 13:45:51.257470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:51.256934    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b" event={"ID":"df482cd7-2dd2-4cb0-b791-e4a9f8c1e097","Type":"ContainerDied","Data":"a76df8b6f7bd182a57d30bef9aee2192bbb02356bd01373ee6e24ecc4582e662"}
Oct 31 13:45:51.257470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:51.257094    4660 scope.go:122] "RemoveContainer" containerID="a407c755e738f29adca841b5c8a767ec8f9b6a43416d8da4c875f0bdacc6d36a"
Oct 31 13:45:51.283360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:51.283336    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"]
Oct 31 13:45:51.295195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:51.295169    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-67k5b"]
Oct 31 13:45:51.456385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:51.456321    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm" podUID="cdc22df5-69c4-4def-9e7d-f022057cd410" containerName="main" containerID="cri-o://e2e232f7db8a59e15bdbeee788152a38f524689440cd17bf45dd72c5922963c4" gracePeriod=45
Oct 31 13:45:51.984074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:51.984043    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:45:52.075477 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.075437    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-tmp-dir\") pod \"cdc22df5-69c4-4def-9e7d-f022057cd410\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") "
Oct 31 13:45:52.075720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.075576    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-model-cache\") pod \"cdc22df5-69c4-4def-9e7d-f022057cd410\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") "
Oct 31 13:45:52.075720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.075712    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdc22df5-69c4-4def-9e7d-f022057cd410-tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc22df5-69c4-4def-9e7d-f022057cd410-tls-certs\") pod \"cdc22df5-69c4-4def-9e7d-f022057cd410\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") "
Oct 31 13:45:52.075873 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.075725    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-tmp-dir" pod "cdc22df5-69c4-4def-9e7d-f022057cd410" (UID: "cdc22df5-69c4-4def-9e7d-f022057cd410"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:52.075873 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.075798    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-dshm\") pod \"cdc22df5-69c4-4def-9e7d-f022057cd410\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") "
Oct 31 13:45:52.076083 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.075894    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-home\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-home\") pod \"cdc22df5-69c4-4def-9e7d-f022057cd410\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") "
Oct 31 13:45:52.076083 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.075910    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-model-cache" pod "cdc22df5-69c4-4def-9e7d-f022057cd410" (UID: "cdc22df5-69c4-4def-9e7d-f022057cd410"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:52.076083 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.075996    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc22df5-69c4-4def-9e7d-f022057cd410-kube-api-access-spjlf\" (UniqueName: \"kubernetes.io/projected/cdc22df5-69c4-4def-9e7d-f022057cd410-kube-api-access-spjlf\") pod \"cdc22df5-69c4-4def-9e7d-f022057cd410\" (UID: \"cdc22df5-69c4-4def-9e7d-f022057cd410\") "
Oct 31 13:45:52.076613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.076276    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-home" pod "cdc22df5-69c4-4def-9e7d-f022057cd410" (UID: "cdc22df5-69c4-4def-9e7d-f022057cd410"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:52.076833 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.076810    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:52.076908 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.076841    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:52.076908 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.076856    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:52.078422 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.078396    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc22df5-69c4-4def-9e7d-f022057cd410-tls-certs" pod "cdc22df5-69c4-4def-9e7d-f022057cd410" (UID: "cdc22df5-69c4-4def-9e7d-f022057cd410"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:45:52.078561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.078534    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-dshm" pod "cdc22df5-69c4-4def-9e7d-f022057cd410" (UID: "cdc22df5-69c4-4def-9e7d-f022057cd410"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:52.078874 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.078847    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc22df5-69c4-4def-9e7d-f022057cd410-kube-api-access-spjlf" pod "cdc22df5-69c4-4def-9e7d-f022057cd410" (UID: "cdc22df5-69c4-4def-9e7d-f022057cd410"). InnerVolumeSpecName "kube-api-access-spjlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:52.178929 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.178880    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc22df5-69c4-4def-9e7d-f022057cd410-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:52.178929 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.178930    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc22df5-69c4-4def-9e7d-f022057cd410-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:52.179140 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.178946    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spjlf\" (UniqueName: \"kubernetes.io/projected/cdc22df5-69c4-4def-9e7d-f022057cd410-kube-api-access-spjlf\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:52.268502 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.268420    4660 generic.go:356] "Generic (PLEG): container finished" podID="cdc22df5-69c4-4def-9e7d-f022057cd410" containerID="e2e232f7db8a59e15bdbeee788152a38f524689440cd17bf45dd72c5922963c4" exitCode=0
Oct 31 13:45:52.268502 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.268472    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm" event={"ID":"cdc22df5-69c4-4def-9e7d-f022057cd410","Type":"ContainerDied","Data":"e2e232f7db8a59e15bdbeee788152a38f524689440cd17bf45dd72c5922963c4"}
Oct 31 13:45:52.268981 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.268514    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"
Oct 31 13:45:52.268981 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.268529    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm" event={"ID":"cdc22df5-69c4-4def-9e7d-f022057cd410","Type":"ContainerDied","Data":"8f4d0285350ca6b165583a22f8b1262e396a1d2794685235cc47cd23aaca51a9"}
Oct 31 13:45:52.268981 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.268568    4660 scope.go:122] "RemoveContainer" containerID="e2e232f7db8a59e15bdbeee788152a38f524689440cd17bf45dd72c5922963c4"
Oct 31 13:45:52.282005 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.281984    4660 scope.go:122] "RemoveContainer" containerID="e2e232f7db8a59e15bdbeee788152a38f524689440cd17bf45dd72c5922963c4"
Oct 31 13:45:52.282392 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:45:52.282370    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e232f7db8a59e15bdbeee788152a38f524689440cd17bf45dd72c5922963c4\": container with ID starting with e2e232f7db8a59e15bdbeee788152a38f524689440cd17bf45dd72c5922963c4 not found: ID does not exist" containerID="e2e232f7db8a59e15bdbeee788152a38f524689440cd17bf45dd72c5922963c4"
Oct 31 13:45:52.282493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.282403    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e232f7db8a59e15bdbeee788152a38f524689440cd17bf45dd72c5922963c4"} err="failed to get container status \"e2e232f7db8a59e15bdbeee788152a38f524689440cd17bf45dd72c5922963c4\": rpc error: code = NotFound desc = could not find container \"e2e232f7db8a59e15bdbeee788152a38f524689440cd17bf45dd72c5922963c4\": container with ID starting with e2e232f7db8a59e15bdbeee788152a38f524689440cd17bf45dd72c5922963c4 not found: ID does not exist"
Oct 31 13:45:52.301739 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.301704    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"]
Oct 31 13:45:52.310748 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.310722    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f4232c/test-model-f4232c-kserve-667644466c-fm7pm"]
Oct 31 13:45:52.530005 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.529925    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc22df5-69c4-4def-9e7d-f022057cd410" path="/var/lib/kubelet/pods/cdc22df5-69c4-4def-9e7d-f022057cd410/volumes"
Oct 31 13:45:52.530454 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:52.530440    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" path="/var/lib/kubelet/pods/df482cd7-2dd2-4cb0-b791-e4a9f8c1e097/volumes"
Oct 31 13:45:54.574122 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:45:54.574082    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"]
Oct 31 13:46:01.143671 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:01.143562    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 13:46:01.157611 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:01.157569    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 13:46:01.199475 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:01.199434    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:46:01.264694 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:01.263561    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:46:01.363926 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:01.363433    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:46:03.225890 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.225830    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j" podUID="740f8e32-74e2-4c64-8d14-e0124a90bd5c" containerName="main" containerID="cri-o://0683390beeea1aa3d67761048320fb291af582c54825a7100dd2294c2739a97d" gracePeriod=45
Oct 31 13:46:03.380195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.380156    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j" podUID="740f8e32-74e2-4c64-8d14-e0124a90bd5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": dial tcp 10.133.0.140:8000: connect: connection refused"
Oct 31 13:46:03.468820 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.468714    4660 generic.go:356] "Generic (PLEG): container finished" podID="740f8e32-74e2-4c64-8d14-e0124a90bd5c" containerID="0683390beeea1aa3d67761048320fb291af582c54825a7100dd2294c2739a97d" exitCode=0
Oct 31 13:46:03.468820 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.468756    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j" event={"ID":"740f8e32-74e2-4c64-8d14-e0124a90bd5c","Type":"ContainerDied","Data":"0683390beeea1aa3d67761048320fb291af582c54825a7100dd2294c2739a97d"}
Oct 31 13:46:03.572382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.572349    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:46:03.683136 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.683102    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-tmp-dir\") pod \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") "
Oct 31 13:46:03.683136 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.683142    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-model-cache\") pod \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") "
Oct 31 13:46:03.683419 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.683289    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-dshm\") pod \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") "
Oct 31 13:46:03.683488 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.683457    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-tmp-dir" pod "740f8e32-74e2-4c64-8d14-e0124a90bd5c" (UID: "740f8e32-74e2-4c64-8d14-e0124a90bd5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:03.683549 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.683463    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/740f8e32-74e2-4c64-8d14-e0124a90bd5c-tls-certs\" (UniqueName: \"kubernetes.io/secret/740f8e32-74e2-4c64-8d14-e0124a90bd5c-tls-certs\") pod \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") "
Oct 31 13:46:03.683549 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.683526    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-model-cache" pod "740f8e32-74e2-4c64-8d14-e0124a90bd5c" (UID: "740f8e32-74e2-4c64-8d14-e0124a90bd5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:03.683671 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.683629    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-home\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-home\") pod \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") "
Oct 31 13:46:03.683724 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.683693    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/740f8e32-74e2-4c64-8d14-e0124a90bd5c-kube-api-access-gqjjb\" (UniqueName: \"kubernetes.io/projected/740f8e32-74e2-4c64-8d14-e0124a90bd5c-kube-api-access-gqjjb\") pod \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\" (UID: \"740f8e32-74e2-4c64-8d14-e0124a90bd5c\") "
Oct 31 13:46:03.683972 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.683947    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-home" pod "740f8e32-74e2-4c64-8d14-e0124a90bd5c" (UID: "740f8e32-74e2-4c64-8d14-e0124a90bd5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:03.684627 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.684602    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:03.684747 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.684663    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:03.684747 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.684679    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:03.685986 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.685957    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/740f8e32-74e2-4c64-8d14-e0124a90bd5c-tls-certs" pod "740f8e32-74e2-4c64-8d14-e0124a90bd5c" (UID: "740f8e32-74e2-4c64-8d14-e0124a90bd5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:46:03.686114 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.686096    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740f8e32-74e2-4c64-8d14-e0124a90bd5c-kube-api-access-gqjjb" pod "740f8e32-74e2-4c64-8d14-e0124a90bd5c" (UID: "740f8e32-74e2-4c64-8d14-e0124a90bd5c"). InnerVolumeSpecName "kube-api-access-gqjjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:03.686661 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.686636    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-dshm" pod "740f8e32-74e2-4c64-8d14-e0124a90bd5c" (UID: "740f8e32-74e2-4c64-8d14-e0124a90bd5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:03.786682 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.786587    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqjjb\" (UniqueName: \"kubernetes.io/projected/740f8e32-74e2-4c64-8d14-e0124a90bd5c-kube-api-access-gqjjb\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:03.786682 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.786618    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/740f8e32-74e2-4c64-8d14-e0124a90bd5c-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:03.786682 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:03.786629    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/740f8e32-74e2-4c64-8d14-e0124a90bd5c-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:04.481508 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:04.481478    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"
Oct 31 13:46:04.481994 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:04.481517    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j" event={"ID":"740f8e32-74e2-4c64-8d14-e0124a90bd5c","Type":"ContainerDied","Data":"6c3cd6139b6b2c3030fa7effc9b863232c875159becf006bf6fe9f189d6ed896"}
Oct 31 13:46:04.481994 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:04.481583    4660 scope.go:122] "RemoveContainer" containerID="0683390beeea1aa3d67761048320fb291af582c54825a7100dd2294c2739a97d"
Oct 31 13:46:04.555316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:04.554071    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"]
Oct 31 13:46:04.560451 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:04.560413    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cae9ec/test-model-cae9ec-kserve-74f7f9945f-7795j"]
Oct 31 13:46:05.819319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:05.818430    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"]
Oct 31 13:46:05.819319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:05.818775    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q" podUID="19742f9a-ed37-4745-a0ac-af1d842a334d" containerName="maas-api" containerID="cri-o://95c465630ac63352b83a99ba31ea4c5f82b9d68a7a8fe2f2ea5feddb9942af1b" gracePeriod=30
Oct 31 13:46:06.639384 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:06.638658    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="740f8e32-74e2-4c64-8d14-e0124a90bd5c" path="/var/lib/kubelet/pods/740f8e32-74e2-4c64-8d14-e0124a90bd5c/volumes"
Oct 31 13:46:06.653339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:06.652214    4660 generic.go:356] "Generic (PLEG): container finished" podID="19742f9a-ed37-4745-a0ac-af1d842a334d" containerID="95c465630ac63352b83a99ba31ea4c5f82b9d68a7a8fe2f2ea5feddb9942af1b" exitCode=0
Oct 31 13:46:06.653339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:06.652434    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q" event={"ID":"19742f9a-ed37-4745-a0ac-af1d842a334d","Type":"ContainerDied","Data":"95c465630ac63352b83a99ba31ea4c5f82b9d68a7a8fe2f2ea5feddb9942af1b"}
Oct 31 13:46:06.723630 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:06.723597    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:46:06.780351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:06.779496    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19742f9a-ed37-4745-a0ac-af1d842a334d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19742f9a-ed37-4745-a0ac-af1d842a334d-openshift-service-ca\") pod \"19742f9a-ed37-4745-a0ac-af1d842a334d\" (UID: \"19742f9a-ed37-4745-a0ac-af1d842a334d\") "
Oct 31 13:46:06.780351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:06.779658    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19742f9a-ed37-4745-a0ac-af1d842a334d-kube-api-access-qxlf5\" (UniqueName: \"kubernetes.io/projected/19742f9a-ed37-4745-a0ac-af1d842a334d-kube-api-access-qxlf5\") pod \"19742f9a-ed37-4745-a0ac-af1d842a334d\" (UID: \"19742f9a-ed37-4745-a0ac-af1d842a334d\") "
Oct 31 13:46:06.780351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:06.779750    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19742f9a-ed37-4745-a0ac-af1d842a334d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/19742f9a-ed37-4745-a0ac-af1d842a334d-metrics-certs\") pod \"19742f9a-ed37-4745-a0ac-af1d842a334d\" (UID: \"19742f9a-ed37-4745-a0ac-af1d842a334d\") "
Oct 31 13:46:06.780351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:06.779828    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19742f9a-ed37-4745-a0ac-af1d842a334d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19742f9a-ed37-4745-a0ac-af1d842a334d-maas-api-tls\") pod \"19742f9a-ed37-4745-a0ac-af1d842a334d\" (UID: \"19742f9a-ed37-4745-a0ac-af1d842a334d\") "
Oct 31 13:46:06.784712 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:06.784680    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19742f9a-ed37-4745-a0ac-af1d842a334d-openshift-service-ca" pod "19742f9a-ed37-4745-a0ac-af1d842a334d" (UID: "19742f9a-ed37-4745-a0ac-af1d842a334d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:06.806664 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:06.806589    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19742f9a-ed37-4745-a0ac-af1d842a334d-maas-api-tls" pod "19742f9a-ed37-4745-a0ac-af1d842a334d" (UID: "19742f9a-ed37-4745-a0ac-af1d842a334d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:46:06.806992 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:06.806939    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19742f9a-ed37-4745-a0ac-af1d842a334d-kube-api-access-qxlf5" pod "19742f9a-ed37-4745-a0ac-af1d842a334d" (UID: "19742f9a-ed37-4745-a0ac-af1d842a334d"). InnerVolumeSpecName "kube-api-access-qxlf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:06.814745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:06.814714    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19742f9a-ed37-4745-a0ac-af1d842a334d-metrics-certs" pod "19742f9a-ed37-4745-a0ac-af1d842a334d" (UID: "19742f9a-ed37-4745-a0ac-af1d842a334d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:46:06.883138 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:06.883063    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19742f9a-ed37-4745-a0ac-af1d842a334d-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:06.883138 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:06.883092    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19742f9a-ed37-4745-a0ac-af1d842a334d-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:06.883138 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:06.883105    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxlf5\" (UniqueName: \"kubernetes.io/projected/19742f9a-ed37-4745-a0ac-af1d842a334d-kube-api-access-qxlf5\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:06.883138 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:06.883116    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19742f9a-ed37-4745-a0ac-af1d842a334d-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:07.663952 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:07.663920    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"
Oct 31 13:46:07.663952 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:07.663933    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q" event={"ID":"19742f9a-ed37-4745-a0ac-af1d842a334d","Type":"ContainerDied","Data":"a96d3981b0f2d788f70e525caf15866ae3c2c6214170980b41fd91eac01128e2"}
Oct 31 13:46:07.664197 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:07.663998    4660 scope.go:122] "RemoveContainer" containerID="95c465630ac63352b83a99ba31ea4c5f82b9d68a7a8fe2f2ea5feddb9942af1b"
Oct 31 13:46:07.690786 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:07.690758    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"]
Oct 31 13:46:07.695427 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:07.695400    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62778a-6565cb9b45-hpn2q"]
Oct 31 13:46:08.534256 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:08.533878    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19742f9a-ed37-4745-a0ac-af1d842a334d" path="/var/lib/kubelet/pods/19742f9a-ed37-4745-a0ac-af1d842a334d/volumes"
Oct 31 13:46:09.607477 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:09.607440    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4" podUID="708ebdbd-2c8c-4d47-8d03-517e149613d1" containerName="main" containerID="cri-o://5af5d7ce7ce3dc0238d7d601da92eb6817397903f5d626a24b791c1a8b5b8772" gracePeriod=45
Oct 31 13:46:09.695038 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:09.694991    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4" podUID="708ebdbd-2c8c-4d47-8d03-517e149613d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.144:8000/ready\": dial tcp 10.133.0.144:8000: connect: connection refused"
Oct 31 13:46:09.955402 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:09.955363    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:46:10.042850 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.042809    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-home\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-home\") pod \"708ebdbd-2c8c-4d47-8d03-517e149613d1\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") "
Oct 31 13:46:10.043052 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.042865    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708ebdbd-2c8c-4d47-8d03-517e149613d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/708ebdbd-2c8c-4d47-8d03-517e149613d1-tls-certs\") pod \"708ebdbd-2c8c-4d47-8d03-517e149613d1\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") "
Oct 31 13:46:10.043124 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.043102    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-model-cache\") pod \"708ebdbd-2c8c-4d47-8d03-517e149613d1\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") "
Oct 31 13:46:10.043193 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.043153    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-dshm\") pod \"708ebdbd-2c8c-4d47-8d03-517e149613d1\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") "
Oct 31 13:46:10.043193 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.043183    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-tmp-dir\") pod \"708ebdbd-2c8c-4d47-8d03-517e149613d1\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") "
Oct 31 13:46:10.043406 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.043287    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708ebdbd-2c8c-4d47-8d03-517e149613d1-kube-api-access-zdc99\" (UniqueName: \"kubernetes.io/projected/708ebdbd-2c8c-4d47-8d03-517e149613d1-kube-api-access-zdc99\") pod \"708ebdbd-2c8c-4d47-8d03-517e149613d1\" (UID: \"708ebdbd-2c8c-4d47-8d03-517e149613d1\") "
Oct 31 13:46:10.050067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.044611    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-model-cache" pod "708ebdbd-2c8c-4d47-8d03-517e149613d1" (UID: "708ebdbd-2c8c-4d47-8d03-517e149613d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:10.050067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.044839    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-home" pod "708ebdbd-2c8c-4d47-8d03-517e149613d1" (UID: "708ebdbd-2c8c-4d47-8d03-517e149613d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:10.050067 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.049990    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708ebdbd-2c8c-4d47-8d03-517e149613d1-kube-api-access-zdc99" pod "708ebdbd-2c8c-4d47-8d03-517e149613d1" (UID: "708ebdbd-2c8c-4d47-8d03-517e149613d1"). InnerVolumeSpecName "kube-api-access-zdc99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:10.050613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.050574    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708ebdbd-2c8c-4d47-8d03-517e149613d1-tls-certs" pod "708ebdbd-2c8c-4d47-8d03-517e149613d1" (UID: "708ebdbd-2c8c-4d47-8d03-517e149613d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:46:10.050712 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.050685    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-dshm" pod "708ebdbd-2c8c-4d47-8d03-517e149613d1" (UID: "708ebdbd-2c8c-4d47-8d03-517e149613d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:10.051405 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.051368    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-tmp-dir" pod "708ebdbd-2c8c-4d47-8d03-517e149613d1" (UID: "708ebdbd-2c8c-4d47-8d03-517e149613d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:10.145230 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.145196    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:10.145230 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.145227    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708ebdbd-2c8c-4d47-8d03-517e149613d1-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:10.145230 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.145236    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:10.145567 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.145244    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:10.145567 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.145253    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/708ebdbd-2c8c-4d47-8d03-517e149613d1-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:10.145567 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.145262    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdc99\" (UniqueName: \"kubernetes.io/projected/708ebdbd-2c8c-4d47-8d03-517e149613d1-kube-api-access-zdc99\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:10.736138 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.736103    4660 generic.go:356] "Generic (PLEG): container finished" podID="708ebdbd-2c8c-4d47-8d03-517e149613d1" containerID="5af5d7ce7ce3dc0238d7d601da92eb6817397903f5d626a24b791c1a8b5b8772" exitCode=0
Oct 31 13:46:10.736597 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.736147    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"
Oct 31 13:46:10.736597 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.736163    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4" event={"ID":"708ebdbd-2c8c-4d47-8d03-517e149613d1","Type":"ContainerDied","Data":"5af5d7ce7ce3dc0238d7d601da92eb6817397903f5d626a24b791c1a8b5b8772"}
Oct 31 13:46:10.736597 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.736201    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4" event={"ID":"708ebdbd-2c8c-4d47-8d03-517e149613d1","Type":"ContainerDied","Data":"ab163fef4618e3d4f4371181b657636653e6d4eec5fceac40d12e28f5bc9d921"}
Oct 31 13:46:10.736597 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.736223    4660 scope.go:122] "RemoveContainer" containerID="5af5d7ce7ce3dc0238d7d601da92eb6817397903f5d626a24b791c1a8b5b8772"
Oct 31 13:46:10.763040 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.763014    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"]
Oct 31 13:46:10.772182 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.772155    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0d835b/test-model-0d835b-kserve-6c7564f9-mr5c4"]
Oct 31 13:46:10.788469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.788450    4660 scope.go:122] "RemoveContainer" containerID="5af5d7ce7ce3dc0238d7d601da92eb6817397903f5d626a24b791c1a8b5b8772"
Oct 31 13:46:10.788844 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:46:10.788820    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af5d7ce7ce3dc0238d7d601da92eb6817397903f5d626a24b791c1a8b5b8772\": container with ID starting with 5af5d7ce7ce3dc0238d7d601da92eb6817397903f5d626a24b791c1a8b5b8772 not found: ID does not exist" containerID="5af5d7ce7ce3dc0238d7d601da92eb6817397903f5d626a24b791c1a8b5b8772"
Oct 31 13:46:10.788962 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:10.788848    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af5d7ce7ce3dc0238d7d601da92eb6817397903f5d626a24b791c1a8b5b8772"} err="failed to get container status \"5af5d7ce7ce3dc0238d7d601da92eb6817397903f5d626a24b791c1a8b5b8772\": rpc error: code = NotFound desc = could not find container \"5af5d7ce7ce3dc0238d7d601da92eb6817397903f5d626a24b791c1a8b5b8772\": container with ID starting with 5af5d7ce7ce3dc0238d7d601da92eb6817397903f5d626a24b791c1a8b5b8772 not found: ID does not exist"
Oct 31 13:46:12.541998 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:12.541962    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708ebdbd-2c8c-4d47-8d03-517e149613d1" path="/var/lib/kubelet/pods/708ebdbd-2c8c-4d47-8d03-517e149613d1/volumes"
Oct 31 13:46:20.739723 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:46:20.739676    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:46:22.103704 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:22.103654    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"]
Oct 31 13:46:22.104197 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:22.103907    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr" podUID="d4e738ae-b6fd-4c4b-9413-0497e07cbc03" containerName="istio-proxy" containerID="cri-o://fb5e316e3b7ccf232c86fae5cd889376172355f4737c9ec37fe99678a48ca19d" gracePeriod=30
Oct 31 13:46:27.845935 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.845907    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:46:27.936042 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.935948    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istiod-ca-cert\") pod \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") "
Oct 31 13:46:27.936042 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.936040    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-data\") pod \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") "
Oct 31 13:46:27.936313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.936123    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-workload-socket\") pod \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") "
Oct 31 13:46:27.936313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.936148    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-token\" (UniqueName: \"kubernetes.io/projected/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-token\") pod \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") "
Oct 31 13:46:27.936313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.936241    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-podinfo\") pod \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") "
Oct 31 13:46:27.936313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.936273    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-data" pod "d4e738ae-b6fd-4c4b-9413-0497e07cbc03" (UID: "d4e738ae-b6fd-4c4b-9413-0497e07cbc03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:27.936560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.936370    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-kube-api-access-lcj44\" (UniqueName: \"kubernetes.io/projected/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-kube-api-access-lcj44\") pod \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") "
Oct 31 13:46:27.936560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.936416    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-workload-socket" pod "d4e738ae-b6fd-4c4b-9413-0497e07cbc03" (UID: "d4e738ae-b6fd-4c4b-9413-0497e07cbc03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:27.936560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.936438    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-credential-socket\") pod \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") "
Oct 31 13:46:27.936560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.936448    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istiod-ca-cert" pod "d4e738ae-b6fd-4c4b-9413-0497e07cbc03" (UID: "d4e738ae-b6fd-4c4b-9413-0497e07cbc03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:27.936560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.936527    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-workload-certs\") pod \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") "
Oct 31 13:46:27.936823 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.936609    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-envoy\") pod \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\" (UID: \"d4e738ae-b6fd-4c4b-9413-0497e07cbc03\") "
Oct 31 13:46:27.936823 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.936718    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-credential-socket" pod "d4e738ae-b6fd-4c4b-9413-0497e07cbc03" (UID: "d4e738ae-b6fd-4c4b-9413-0497e07cbc03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:27.937500 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.937253    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-workload-certs" pod "d4e738ae-b6fd-4c4b-9413-0497e07cbc03" (UID: "d4e738ae-b6fd-4c4b-9413-0497e07cbc03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:27.937630 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.937481    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:27.937630 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.937553    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:27.937630 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.937569    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:27.937630 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.937585    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:27.937630 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.937600    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:27.939099 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.939072    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-podinfo" pod "d4e738ae-b6fd-4c4b-9413-0497e07cbc03" (UID: "d4e738ae-b6fd-4c4b-9413-0497e07cbc03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:46:27.939500 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.939473    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-token" pod "d4e738ae-b6fd-4c4b-9413-0497e07cbc03" (UID: "d4e738ae-b6fd-4c4b-9413-0497e07cbc03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:27.939709 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.939689    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-envoy" pod "d4e738ae-b6fd-4c4b-9413-0497e07cbc03" (UID: "d4e738ae-b6fd-4c4b-9413-0497e07cbc03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:27.939941 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:27.939914    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-kube-api-access-lcj44" pod "d4e738ae-b6fd-4c4b-9413-0497e07cbc03" (UID: "d4e738ae-b6fd-4c4b-9413-0497e07cbc03"). InnerVolumeSpecName "kube-api-access-lcj44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:28.011969 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.011366    4660 generic.go:356] "Generic (PLEG): container finished" podID="d4e738ae-b6fd-4c4b-9413-0497e07cbc03" containerID="fb5e316e3b7ccf232c86fae5cd889376172355f4737c9ec37fe99678a48ca19d" exitCode=0
Oct 31 13:46:28.011969 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.011396    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr" event={"ID":"d4e738ae-b6fd-4c4b-9413-0497e07cbc03","Type":"ContainerDied","Data":"fb5e316e3b7ccf232c86fae5cd889376172355f4737c9ec37fe99678a48ca19d"}
Oct 31 13:46:28.011969 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.011381    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"
Oct 31 13:46:28.011969 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.011501    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr" event={"ID":"d4e738ae-b6fd-4c4b-9413-0497e07cbc03","Type":"ContainerDied","Data":"90cf10152ce03bb3f08346a891e24b4195569b413310e5d073ffb81499fb88d6"}
Oct 31 13:46:28.011969 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.011532    4660 scope.go:122] "RemoveContainer" containerID="fb5e316e3b7ccf232c86fae5cd889376172355f4737c9ec37fe99678a48ca19d"
Oct 31 13:46:28.025234 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.025154    4660 scope.go:122] "RemoveContainer" containerID="1198ab63c389eb8eae6823862d0dd42d6609d985f42eb1d1e591615c7e77060e"
Oct 31 13:46:28.035813 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.035795    4660 scope.go:122] "RemoveContainer" containerID="fb5e316e3b7ccf232c86fae5cd889376172355f4737c9ec37fe99678a48ca19d"
Oct 31 13:46:28.036076 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:46:28.036057    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb5e316e3b7ccf232c86fae5cd889376172355f4737c9ec37fe99678a48ca19d\": container with ID starting with fb5e316e3b7ccf232c86fae5cd889376172355f4737c9ec37fe99678a48ca19d not found: ID does not exist" containerID="fb5e316e3b7ccf232c86fae5cd889376172355f4737c9ec37fe99678a48ca19d"
Oct 31 13:46:28.036121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.036086    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb5e316e3b7ccf232c86fae5cd889376172355f4737c9ec37fe99678a48ca19d"} err="failed to get container status \"fb5e316e3b7ccf232c86fae5cd889376172355f4737c9ec37fe99678a48ca19d\": rpc error: code = NotFound desc = could not find container \"fb5e316e3b7ccf232c86fae5cd889376172355f4737c9ec37fe99678a48ca19d\": container with ID starting with fb5e316e3b7ccf232c86fae5cd889376172355f4737c9ec37fe99678a48ca19d not found: ID does not exist"
Oct 31 13:46:28.036121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.036104    4660 scope.go:122] "RemoveContainer" containerID="1198ab63c389eb8eae6823862d0dd42d6609d985f42eb1d1e591615c7e77060e"
Oct 31 13:46:28.036388 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:46:28.036365    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1198ab63c389eb8eae6823862d0dd42d6609d985f42eb1d1e591615c7e77060e\": container with ID starting with 1198ab63c389eb8eae6823862d0dd42d6609d985f42eb1d1e591615c7e77060e not found: ID does not exist" containerID="1198ab63c389eb8eae6823862d0dd42d6609d985f42eb1d1e591615c7e77060e"
Oct 31 13:46:28.036907 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.036396    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1198ab63c389eb8eae6823862d0dd42d6609d985f42eb1d1e591615c7e77060e"} err="failed to get container status \"1198ab63c389eb8eae6823862d0dd42d6609d985f42eb1d1e591615c7e77060e\": rpc error: code = NotFound desc = could not find container \"1198ab63c389eb8eae6823862d0dd42d6609d985f42eb1d1e591615c7e77060e\": container with ID starting with 1198ab63c389eb8eae6823862d0dd42d6609d985f42eb1d1e591615c7e77060e not found: ID does not exist"
Oct 31 13:46:28.039283 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.039263    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:28.039343 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.039286    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:28.039343 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.039312    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:28.039343 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.039321    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcj44\" (UniqueName: \"kubernetes.io/projected/d4e738ae-b6fd-4c4b-9413-0497e07cbc03-kube-api-access-lcj44\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:28.047694 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.047673    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"]
Oct 31 13:46:28.056322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.056284    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-61f295-openshift-default-6797665b47-ws5wr"]
Oct 31 13:46:28.529681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:28.529648    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e738ae-b6fd-4c4b-9413-0497e07cbc03" path="/var/lib/kubelet/pods/d4e738ae-b6fd-4c4b-9413-0497e07cbc03/volumes"
Oct 31 13:46:33.766502 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:33.765487    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"]
Oct 31 13:46:33.766502 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:33.765729    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92" podUID="0e4a9639-2171-4539-abca-aa29d66bcdfb" containerName="istio-proxy" containerID="cri-o://d72aed58664859312761d9239e80d55c974c2f4e2bc54a6ef958b1e9fa3df567" gracePeriod=30
Oct 31 13:46:33.772903 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:33.772588    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"]
Oct 31 13:46:33.772903 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:33.772778    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk" podUID="77c1be44-f679-4953-9a0d-74f8c160e29c" containerName="istio-proxy" containerID="cri-o://3cb8743ac7c7f749484cf0c76811676182f2663853702b79d6b827a01e5448d6" gracePeriod=30
Oct 31 13:46:33.779268 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:33.779233    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"]
Oct 31 13:46:33.779512 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:33.779487    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p" podUID="fef53539-9688-4af7-8ce0-5aa0bbea4f9c" containerName="istio-proxy" containerID="cri-o://83a6a07a9fb726bf2c9b09bf6a1ff33cf38a7e08bf0ebfc423327f3cb66a9036" gracePeriod=30
Oct 31 13:46:36.095618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:36.095579    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:46:36.096198 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:36.095641    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk" podUID="77c1be44-f679-4953-9a0d-74f8c160e29c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:46:36.179462 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:36.179383    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"]
Oct 31 13:46:36.179677 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:36.179644    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj" podUID="239235d5-45ef-405c-8c3a-02618f5ebaa0" containerName="istio-proxy" containerID="cri-o://dca0d9584297e37f794a7abb2dcd6cab6829b6f751396b5a81d2c2ed44b534e9" gracePeriod=30
Oct 31 13:46:36.180781 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:36.180450    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"]
Oct 31 13:46:36.180781 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:36.180624    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6" podUID="fecef293-ce25-4fc8-84cf-91140172b297" containerName="istio-proxy" containerID="cri-o://66382f62e91bbbb7698cc561d8e1d3d88decadb0cf17fa2f01b1951b7fd4b331" gracePeriod=30
Oct 31 13:46:37.208969 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:37.208927    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:46:37.209472 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:37.208999    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92" podUID="0e4a9639-2171-4539-abca-aa29d66bcdfb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:46:37.256944 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:37.256899    4660 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:46:37.257172 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:37.256973    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p" podUID="fef53539-9688-4af7-8ce0-5aa0bbea4f9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:46:39.143573 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.136260    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:46:39.178344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.175763    4660 generic.go:356] "Generic (PLEG): container finished" podID="77c1be44-f679-4953-9a0d-74f8c160e29c" containerID="3cb8743ac7c7f749484cf0c76811676182f2663853702b79d6b827a01e5448d6" exitCode=0
Oct 31 13:46:39.178699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.178660    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk" event={"ID":"77c1be44-f679-4953-9a0d-74f8c160e29c","Type":"ContainerDied","Data":"3cb8743ac7c7f749484cf0c76811676182f2663853702b79d6b827a01e5448d6"}
Oct 31 13:46:39.183131 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.183101    4660 generic.go:356] "Generic (PLEG): container finished" podID="0e4a9639-2171-4539-abca-aa29d66bcdfb" containerID="d72aed58664859312761d9239e80d55c974c2f4e2bc54a6ef958b1e9fa3df567" exitCode=0
Oct 31 13:46:39.183274 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.183189    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92" event={"ID":"0e4a9639-2171-4539-abca-aa29d66bcdfb","Type":"ContainerDied","Data":"d72aed58664859312761d9239e80d55c974c2f4e2bc54a6ef958b1e9fa3df567"}
Oct 31 13:46:39.183274 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.183226    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92" event={"ID":"0e4a9639-2171-4539-abca-aa29d66bcdfb","Type":"ContainerDied","Data":"195e928d1c705b184b34c73e9c5ae1918cfcb00128594503c767f896ff1b044c"}
Oct 31 13:46:39.183274 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.183245    4660 scope.go:122] "RemoveContainer" containerID="d72aed58664859312761d9239e80d55c974c2f4e2bc54a6ef958b1e9fa3df567"
Oct 31 13:46:39.183493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.183397    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"
Oct 31 13:46:39.188032 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.188001    4660 generic.go:356] "Generic (PLEG): container finished" podID="fef53539-9688-4af7-8ce0-5aa0bbea4f9c" containerID="83a6a07a9fb726bf2c9b09bf6a1ff33cf38a7e08bf0ebfc423327f3cb66a9036" exitCode=0
Oct 31 13:46:39.188153 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.188056    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p" event={"ID":"fef53539-9688-4af7-8ce0-5aa0bbea4f9c","Type":"ContainerDied","Data":"83a6a07a9fb726bf2c9b09bf6a1ff33cf38a7e08bf0ebfc423327f3cb66a9036"}
Oct 31 13:46:39.190696 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.190671    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:46:39.193509 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.193486    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:46:39.202425 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.202333    4660 scope.go:122] "RemoveContainer" containerID="d72aed58664859312761d9239e80d55c974c2f4e2bc54a6ef958b1e9fa3df567"
Oct 31 13:46:39.202753 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:46:39.202728    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d72aed58664859312761d9239e80d55c974c2f4e2bc54a6ef958b1e9fa3df567\": container with ID starting with d72aed58664859312761d9239e80d55c974c2f4e2bc54a6ef958b1e9fa3df567 not found: ID does not exist" containerID="d72aed58664859312761d9239e80d55c974c2f4e2bc54a6ef958b1e9fa3df567"
Oct 31 13:46:39.202891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.202761    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d72aed58664859312761d9239e80d55c974c2f4e2bc54a6ef958b1e9fa3df567"} err="failed to get container status \"d72aed58664859312761d9239e80d55c974c2f4e2bc54a6ef958b1e9fa3df567\": rpc error: code = NotFound desc = could not find container \"d72aed58664859312761d9239e80d55c974c2f4e2bc54a6ef958b1e9fa3df567\": container with ID starting with d72aed58664859312761d9239e80d55c974c2f4e2bc54a6ef958b1e9fa3df567 not found: ID does not exist"
Oct 31 13:46:39.243762 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.243728    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-workload-socket\") pod \"0e4a9639-2171-4539-abca-aa29d66bcdfb\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") "
Oct 31 13:46:39.243963 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.243837    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e4a9639-2171-4539-abca-aa29d66bcdfb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4a9639-2171-4539-abca-aa29d66bcdfb-istiod-ca-cert\") pod \"0e4a9639-2171-4539-abca-aa29d66bcdfb\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") "
Oct 31 13:46:39.243963 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.243885    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-podinfo\") pod \"0e4a9639-2171-4539-abca-aa29d66bcdfb\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") "
Oct 31 13:46:39.243963 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.243948    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-data\") pod \"0e4a9639-2171-4539-abca-aa29d66bcdfb\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") "
Oct 31 13:46:39.244205 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.244016    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-workload-certs\") pod \"0e4a9639-2171-4539-abca-aa29d66bcdfb\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") "
Oct 31 13:46:39.244205 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.244030    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-workload-socket" pod "0e4a9639-2171-4539-abca-aa29d66bcdfb" (UID: "0e4a9639-2171-4539-abca-aa29d66bcdfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.244205 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.244132    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4a9639-2171-4539-abca-aa29d66bcdfb-kube-api-access-ccmpx\" (UniqueName: \"kubernetes.io/projected/0e4a9639-2171-4539-abca-aa29d66bcdfb-kube-api-access-ccmpx\") pod \"0e4a9639-2171-4539-abca-aa29d66bcdfb\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") "
Oct 31 13:46:39.244205 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.244186    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-credential-socket\") pod \"0e4a9639-2171-4539-abca-aa29d66bcdfb\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") "
Oct 31 13:46:39.244458 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.244245    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-token\" (UniqueName: \"kubernetes.io/projected/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-token\") pod \"0e4a9639-2171-4539-abca-aa29d66bcdfb\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") "
Oct 31 13:46:39.244458 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.244261    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4a9639-2171-4539-abca-aa29d66bcdfb-istiod-ca-cert" pod "0e4a9639-2171-4539-abca-aa29d66bcdfb" (UID: "0e4a9639-2171-4539-abca-aa29d66bcdfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:39.244562 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.244486    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-envoy\") pod \"0e4a9639-2171-4539-abca-aa29d66bcdfb\" (UID: \"0e4a9639-2171-4539-abca-aa29d66bcdfb\") "
Oct 31 13:46:39.244847 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.244817    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-workload-certs" pod "0e4a9639-2171-4539-abca-aa29d66bcdfb" (UID: "0e4a9639-2171-4539-abca-aa29d66bcdfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.245075 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.245051    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-data" pod "0e4a9639-2171-4539-abca-aa29d66bcdfb" (UID: "0e4a9639-2171-4539-abca-aa29d66bcdfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.245529 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.245461    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.245529 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.245489    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.245529 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.245503    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4a9639-2171-4539-abca-aa29d66bcdfb-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.245529 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.245516    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.246029 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.245755    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-credential-socket" pod "0e4a9639-2171-4539-abca-aa29d66bcdfb" (UID: "0e4a9639-2171-4539-abca-aa29d66bcdfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.248066 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.247951    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4a9639-2171-4539-abca-aa29d66bcdfb-kube-api-access-ccmpx" pod "0e4a9639-2171-4539-abca-aa29d66bcdfb" (UID: "0e4a9639-2171-4539-abca-aa29d66bcdfb"). InnerVolumeSpecName "kube-api-access-ccmpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:39.248283 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.248172    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-token" pod "0e4a9639-2171-4539-abca-aa29d66bcdfb" (UID: "0e4a9639-2171-4539-abca-aa29d66bcdfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:39.248532 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.248502    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-envoy" pod "0e4a9639-2171-4539-abca-aa29d66bcdfb" (UID: "0e4a9639-2171-4539-abca-aa29d66bcdfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.257368 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.257332    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-podinfo" pod "0e4a9639-2171-4539-abca-aa29d66bcdfb" (UID: "0e4a9639-2171-4539-abca-aa29d66bcdfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:46:39.346278 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346201    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istiod-ca-cert\") pod \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") "
Oct 31 13:46:39.346278 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346263    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-credential-socket\") pod \"77c1be44-f679-4953-9a0d-74f8c160e29c\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") "
Oct 31 13:46:39.346560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346365    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c1be44-f679-4953-9a0d-74f8c160e29c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c1be44-f679-4953-9a0d-74f8c160e29c-istiod-ca-cert\") pod \"77c1be44-f679-4953-9a0d-74f8c160e29c\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") "
Oct 31 13:46:39.346560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346409    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-data\") pod \"77c1be44-f679-4953-9a0d-74f8c160e29c\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") "
Oct 31 13:46:39.346560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346461    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-data\") pod \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") "
Oct 31 13:46:39.346560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346519    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-credential-socket\") pod \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") "
Oct 31 13:46:39.346777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346557    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-credential-socket" pod "77c1be44-f679-4953-9a0d-74f8c160e29c" (UID: "77c1be44-f679-4953-9a0d-74f8c160e29c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.346777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346565    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-podinfo\") pod \"77c1be44-f679-4953-9a0d-74f8c160e29c\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") "
Oct 31 13:46:39.346777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346683    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-token\" (UniqueName: \"kubernetes.io/projected/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-token\") pod \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") "
Oct 31 13:46:39.346943 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346781    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-data" pod "fef53539-9688-4af7-8ce0-5aa0bbea4f9c" (UID: "fef53539-9688-4af7-8ce0-5aa0bbea4f9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.346943 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346785    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-token\" (UniqueName: \"kubernetes.io/projected/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-token\") pod \"77c1be44-f679-4953-9a0d-74f8c160e29c\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") "
Oct 31 13:46:39.346943 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346780    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istiod-ca-cert" pod "fef53539-9688-4af7-8ce0-5aa0bbea4f9c" (UID: "fef53539-9688-4af7-8ce0-5aa0bbea4f9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:39.346943 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346859    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-workload-socket\") pod \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") "
Oct 31 13:46:39.346943 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346835    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-data" pod "77c1be44-f679-4953-9a0d-74f8c160e29c" (UID: "77c1be44-f679-4953-9a0d-74f8c160e29c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.347229 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346869    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-credential-socket" pod "fef53539-9688-4af7-8ce0-5aa0bbea4f9c" (UID: "fef53539-9688-4af7-8ce0-5aa0bbea4f9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.347229 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346955    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-workload-certs\") pod \"77c1be44-f679-4953-9a0d-74f8c160e29c\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") "
Oct 31 13:46:39.347229 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.346991    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-envoy\") pod \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") "
Oct 31 13:46:39.347229 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.347020    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-kube-api-access-lkllf\" (UniqueName: \"kubernetes.io/projected/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-kube-api-access-lkllf\") pod \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") "
Oct 31 13:46:39.347229 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.347109    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-workload-socket" pod "fef53539-9688-4af7-8ce0-5aa0bbea4f9c" (UID: "fef53539-9688-4af7-8ce0-5aa0bbea4f9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.347229 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.347134    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-workload-certs" pod "77c1be44-f679-4953-9a0d-74f8c160e29c" (UID: "77c1be44-f679-4953-9a0d-74f8c160e29c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.347229 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.347178    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c1be44-f679-4953-9a0d-74f8c160e29c-istiod-ca-cert" pod "77c1be44-f679-4953-9a0d-74f8c160e29c" (UID: "77c1be44-f679-4953-9a0d-74f8c160e29c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:39.347871 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.347421    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-workload-socket\") pod \"77c1be44-f679-4953-9a0d-74f8c160e29c\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") "
Oct 31 13:46:39.347871 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.347531    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-podinfo\") pod \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") "
Oct 31 13:46:39.347871 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.347575    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-workload-certs\") pod \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\" (UID: \"fef53539-9688-4af7-8ce0-5aa0bbea4f9c\") "
Oct 31 13:46:39.347871 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.347662    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c1be44-f679-4953-9a0d-74f8c160e29c-kube-api-access-bt85v\" (UniqueName: \"kubernetes.io/projected/77c1be44-f679-4953-9a0d-74f8c160e29c-kube-api-access-bt85v\") pod \"77c1be44-f679-4953-9a0d-74f8c160e29c\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") "
Oct 31 13:46:39.347871 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.347703    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-envoy\") pod \"77c1be44-f679-4953-9a0d-74f8c160e29c\" (UID: \"77c1be44-f679-4953-9a0d-74f8c160e29c\") "
Oct 31 13:46:39.348261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.348237    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-workload-socket" pod "77c1be44-f679-4953-9a0d-74f8c160e29c" (UID: "77c1be44-f679-4953-9a0d-74f8c160e29c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.348515 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.348493    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.348515 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.348516    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c1be44-f679-4953-9a0d-74f8c160e29c-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.348664 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.348532    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccmpx\" (UniqueName: \"kubernetes.io/projected/0e4a9639-2171-4539-abca-aa29d66bcdfb-kube-api-access-ccmpx\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.348664 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.348546    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.348664 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.348560    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.348664 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.348576    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.348664 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.348589    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.348664 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.348603    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.348664 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.348617    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.348664 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.348649    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.348664 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.348663    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.349156 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.348700    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.349156 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.348715    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4a9639-2171-4539-abca-aa29d66bcdfb-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.349156 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.348730    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.349743 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.349668    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-workload-certs" pod "fef53539-9688-4af7-8ce0-5aa0bbea4f9c" (UID: "fef53539-9688-4af7-8ce0-5aa0bbea4f9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.349743 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.349735    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-token" pod "77c1be44-f679-4953-9a0d-74f8c160e29c" (UID: "77c1be44-f679-4953-9a0d-74f8c160e29c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:39.349938 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.349771    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-kube-api-access-lkllf" pod "fef53539-9688-4af7-8ce0-5aa0bbea4f9c" (UID: "fef53539-9688-4af7-8ce0-5aa0bbea4f9c"). InnerVolumeSpecName "kube-api-access-lkllf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:39.350133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.350102    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-podinfo" pod "77c1be44-f679-4953-9a0d-74f8c160e29c" (UID: "77c1be44-f679-4953-9a0d-74f8c160e29c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:46:39.350470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.350440    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-envoy" pod "fef53539-9688-4af7-8ce0-5aa0bbea4f9c" (UID: "fef53539-9688-4af7-8ce0-5aa0bbea4f9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.350598 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.350545    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-token" pod "fef53539-9688-4af7-8ce0-5aa0bbea4f9c" (UID: "fef53539-9688-4af7-8ce0-5aa0bbea4f9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:39.351034 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.351017    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-envoy" pod "77c1be44-f679-4953-9a0d-74f8c160e29c" (UID: "77c1be44-f679-4953-9a0d-74f8c160e29c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.351362 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.351343    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c1be44-f679-4953-9a0d-74f8c160e29c-kube-api-access-bt85v" pod "77c1be44-f679-4953-9a0d-74f8c160e29c" (UID: "77c1be44-f679-4953-9a0d-74f8c160e29c"). InnerVolumeSpecName "kube-api-access-bt85v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:39.351504 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.351483    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-podinfo" pod "fef53539-9688-4af7-8ce0-5aa0bbea4f9c" (UID: "fef53539-9688-4af7-8ce0-5aa0bbea4f9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:46:39.450680 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.450646    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.450680 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.450675    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.450680 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.450685    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.450980 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.450695    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.450980 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.450704    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkllf\" (UniqueName: \"kubernetes.io/projected/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-kube-api-access-lkllf\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.450980 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.450714    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.450980 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.450725    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef53539-9688-4af7-8ce0-5aa0bbea4f9c-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.450980 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.450734    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt85v\" (UniqueName: \"kubernetes.io/projected/77c1be44-f679-4953-9a0d-74f8c160e29c-kube-api-access-bt85v\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.450980 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.450742    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c1be44-f679-4953-9a0d-74f8c160e29c-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.516986 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.516955    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"]
Oct 31 13:46:39.523902 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:39.523876    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jwg92"]
Oct 31 13:46:40.199301 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:40.199262    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p" event={"ID":"fef53539-9688-4af7-8ce0-5aa0bbea4f9c","Type":"ContainerDied","Data":"c063f362ef1bb99dfaae8bdba72ba61731d19a6ce94ba411b5a11f96769596a1"}
Oct 31 13:46:40.199804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:40.199243    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"
Oct 31 13:46:40.199804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:40.199349    4660 scope.go:122] "RemoveContainer" containerID="83a6a07a9fb726bf2c9b09bf6a1ff33cf38a7e08bf0ebfc423327f3cb66a9036"
Oct 31 13:46:40.201452 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:40.201412    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"
Oct 31 13:46:40.201745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:40.201674    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk" event={"ID":"77c1be44-f679-4953-9a0d-74f8c160e29c","Type":"ContainerDied","Data":"89dea5bdf4d90a88c180c2219d7c44cc20aac77fc6aca764a6998a5def99677e"}
Oct 31 13:46:40.214887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:40.214865    4660 scope.go:122] "RemoveContainer" containerID="3cb8743ac7c7f749484cf0c76811676182f2663853702b79d6b827a01e5448d6"
Oct 31 13:46:40.242168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:40.242132    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"]
Oct 31 13:46:40.250581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:40.250557    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-7ntgk"]
Oct 31 13:46:40.277256 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:40.277222    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"]
Oct 31 13:46:40.282365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:40.282342    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-jvt9p"]
Oct 31 13:46:40.529156 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:40.529069    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e4a9639-2171-4539-abca-aa29d66bcdfb" path="/var/lib/kubelet/pods/0e4a9639-2171-4539-abca-aa29d66bcdfb/volumes"
Oct 31 13:46:40.529793 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:40.529767    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c1be44-f679-4953-9a0d-74f8c160e29c" path="/var/lib/kubelet/pods/77c1be44-f679-4953-9a0d-74f8c160e29c/volumes"
Oct 31 13:46:40.530360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:40.530339    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef53539-9688-4af7-8ce0-5aa0bbea4f9c" path="/var/lib/kubelet/pods/fef53539-9688-4af7-8ce0-5aa0bbea4f9c/volumes"
Oct 31 13:46:41.540873 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.540836    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:46:41.562384 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.562354    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:46:41.702210 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702176    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-credential-socket\") pod \"239235d5-45ef-405c-8c3a-02618f5ebaa0\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") "
Oct 31 13:46:41.702446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702238    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-envoy\") pod \"239235d5-45ef-405c-8c3a-02618f5ebaa0\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") "
Oct 31 13:46:41.702446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702263    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fecef293-ce25-4fc8-84cf-91140172b297-kube-api-access-hqtlr\" (UniqueName: \"kubernetes.io/projected/fecef293-ce25-4fc8-84cf-91140172b297-kube-api-access-hqtlr\") pod \"fecef293-ce25-4fc8-84cf-91140172b297\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") "
Oct 31 13:46:41.702446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702325    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239235d5-45ef-405c-8c3a-02618f5ebaa0-kube-api-access-rjp79\" (UniqueName: \"kubernetes.io/projected/239235d5-45ef-405c-8c3a-02618f5ebaa0-kube-api-access-rjp79\") pod \"239235d5-45ef-405c-8c3a-02618f5ebaa0\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") "
Oct 31 13:46:41.702446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702352    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-data\") pod \"239235d5-45ef-405c-8c3a-02618f5ebaa0\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") "
Oct 31 13:46:41.702446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702377    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fecef293-ce25-4fc8-84cf-91140172b297-istio-token\" (UniqueName: \"kubernetes.io/projected/fecef293-ce25-4fc8-84cf-91140172b297-istio-token\") pod \"fecef293-ce25-4fc8-84cf-91140172b297\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") "
Oct 31 13:46:41.702446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702410    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fecef293-ce25-4fc8-84cf-91140172b297-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecef293-ce25-4fc8-84cf-91140172b297-istio-podinfo\") pod \"fecef293-ce25-4fc8-84cf-91140172b297\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") "
Oct 31 13:46:41.702764 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702464    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-workload-certs\") pod \"239235d5-45ef-405c-8c3a-02618f5ebaa0\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") "
Oct 31 13:46:41.702764 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702494    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-istio-data\") pod \"fecef293-ce25-4fc8-84cf-91140172b297\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") "
Oct 31 13:46:41.702764 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702553    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/239235d5-45ef-405c-8c3a-02618f5ebaa0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239235d5-45ef-405c-8c3a-02618f5ebaa0-istiod-ca-cert\") pod \"239235d5-45ef-405c-8c3a-02618f5ebaa0\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") "
Oct 31 13:46:41.702764 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702596    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-istio-envoy\") pod \"fecef293-ce25-4fc8-84cf-91140172b297\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") "
Oct 31 13:46:41.702764 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702612    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fecef293-ce25-4fc8-84cf-91140172b297-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecef293-ce25-4fc8-84cf-91140172b297-istiod-ca-cert\") pod \"fecef293-ce25-4fc8-84cf-91140172b297\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") "
Oct 31 13:46:41.702764 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702656    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-credential-socket\") pod \"fecef293-ce25-4fc8-84cf-91140172b297\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") "
Oct 31 13:46:41.702764 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702695    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-workload-certs\") pod \"fecef293-ce25-4fc8-84cf-91140172b297\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") "
Oct 31 13:46:41.702764 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702728    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-workload-socket\") pod \"fecef293-ce25-4fc8-84cf-91140172b297\" (UID: \"fecef293-ce25-4fc8-84cf-91140172b297\") "
Oct 31 13:46:41.702764 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702746    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-workload-socket\") pod \"239235d5-45ef-405c-8c3a-02618f5ebaa0\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") "
Oct 31 13:46:41.703195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702783    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-podinfo\") pod \"239235d5-45ef-405c-8c3a-02618f5ebaa0\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") "
Oct 31 13:46:41.703195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.702826    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-token\" (UniqueName: \"kubernetes.io/projected/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-token\") pod \"239235d5-45ef-405c-8c3a-02618f5ebaa0\" (UID: \"239235d5-45ef-405c-8c3a-02618f5ebaa0\") "
Oct 31 13:46:41.704022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.703847    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-credential-socket" pod "239235d5-45ef-405c-8c3a-02618f5ebaa0" (UID: "239235d5-45ef-405c-8c3a-02618f5ebaa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.704022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.703933    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-workload-certs" pod "fecef293-ce25-4fc8-84cf-91140172b297" (UID: "fecef293-ce25-4fc8-84cf-91140172b297"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.704917 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.704236    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-workload-socket" pod "fecef293-ce25-4fc8-84cf-91140172b297" (UID: "fecef293-ce25-4fc8-84cf-91140172b297"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.704917 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.704448    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-data" pod "239235d5-45ef-405c-8c3a-02618f5ebaa0" (UID: "239235d5-45ef-405c-8c3a-02618f5ebaa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.704917 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.704536    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fecef293-ce25-4fc8-84cf-91140172b297-istiod-ca-cert" pod "fecef293-ce25-4fc8-84cf-91140172b297" (UID: "fecef293-ce25-4fc8-84cf-91140172b297"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:41.704917 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.704658    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-credential-socket" pod "fecef293-ce25-4fc8-84cf-91140172b297" (UID: "fecef293-ce25-4fc8-84cf-91140172b297"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.704917 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.704820    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-workload-socket" pod "239235d5-45ef-405c-8c3a-02618f5ebaa0" (UID: "239235d5-45ef-405c-8c3a-02618f5ebaa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.705333 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.705311    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-workload-certs" pod "239235d5-45ef-405c-8c3a-02618f5ebaa0" (UID: "239235d5-45ef-405c-8c3a-02618f5ebaa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.705801 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.705583    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-istio-data" pod "fecef293-ce25-4fc8-84cf-91140172b297" (UID: "fecef293-ce25-4fc8-84cf-91140172b297"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.705916 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.705890    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/239235d5-45ef-405c-8c3a-02618f5ebaa0-istiod-ca-cert" pod "239235d5-45ef-405c-8c3a-02618f5ebaa0" (UID: "239235d5-45ef-405c-8c3a-02618f5ebaa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:41.715349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.712840    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fecef293-ce25-4fc8-84cf-91140172b297-kube-api-access-hqtlr" pod "fecef293-ce25-4fc8-84cf-91140172b297" (UID: "fecef293-ce25-4fc8-84cf-91140172b297"). InnerVolumeSpecName "kube-api-access-hqtlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:41.715349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.713017    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-token" pod "239235d5-45ef-405c-8c3a-02618f5ebaa0" (UID: "239235d5-45ef-405c-8c3a-02618f5ebaa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:41.715497 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.715407    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-envoy" pod "239235d5-45ef-405c-8c3a-02618f5ebaa0" (UID: "239235d5-45ef-405c-8c3a-02618f5ebaa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.715650 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.715630    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fecef293-ce25-4fc8-84cf-91140172b297-istio-podinfo" pod "fecef293-ce25-4fc8-84cf-91140172b297" (UID: "fecef293-ce25-4fc8-84cf-91140172b297"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:46:41.715750 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.715726    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-podinfo" pod "239235d5-45ef-405c-8c3a-02618f5ebaa0" (UID: "239235d5-45ef-405c-8c3a-02618f5ebaa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:46:41.715902 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.715808    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fecef293-ce25-4fc8-84cf-91140172b297-istio-token" pod "fecef293-ce25-4fc8-84cf-91140172b297" (UID: "fecef293-ce25-4fc8-84cf-91140172b297"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:41.715902 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.715866    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-istio-envoy" pod "fecef293-ce25-4fc8-84cf-91140172b297" (UID: "fecef293-ce25-4fc8-84cf-91140172b297"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.715902 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.715866    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239235d5-45ef-405c-8c3a-02618f5ebaa0-kube-api-access-rjp79" pod "239235d5-45ef-405c-8c3a-02618f5ebaa0" (UID: "239235d5-45ef-405c-8c3a-02618f5ebaa0"). InnerVolumeSpecName "kube-api-access-rjp79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808597    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808636    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808649    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808662    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808673    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808685    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808696    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808707    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808719    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqtlr\" (UniqueName: \"kubernetes.io/projected/fecef293-ce25-4fc8-84cf-91140172b297-kube-api-access-hqtlr\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808733    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjp79\" (UniqueName: \"kubernetes.io/projected/239235d5-45ef-405c-8c3a-02618f5ebaa0-kube-api-access-rjp79\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808746    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808758    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecef293-ce25-4fc8-84cf-91140172b297-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808770    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecef293-ce25-4fc8-84cf-91140172b297-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808781    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239235d5-45ef-405c-8c3a-02618f5ebaa0-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808792    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808804    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239235d5-45ef-405c-8c3a-02618f5ebaa0-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808815    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecef293-ce25-4fc8-84cf-91140172b297-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.808913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:41.808829    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecef293-ce25-4fc8-84cf-91140172b297-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:42.229505 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.228641    4660 generic.go:356] "Generic (PLEG): container finished" podID="239235d5-45ef-405c-8c3a-02618f5ebaa0" containerID="dca0d9584297e37f794a7abb2dcd6cab6829b6f751396b5a81d2c2ed44b534e9" exitCode=0
Oct 31 13:46:42.229505 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.228701    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"
Oct 31 13:46:42.229505 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.228725    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj" event={"ID":"239235d5-45ef-405c-8c3a-02618f5ebaa0","Type":"ContainerDied","Data":"dca0d9584297e37f794a7abb2dcd6cab6829b6f751396b5a81d2c2ed44b534e9"}
Oct 31 13:46:42.229505 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.228791    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj" event={"ID":"239235d5-45ef-405c-8c3a-02618f5ebaa0","Type":"ContainerDied","Data":"8ca7c9c50b911a9d275cd847eb3f44c39d11da72995696adbc15c6a25aaba25d"}
Oct 31 13:46:42.229505 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.228820    4660 scope.go:122] "RemoveContainer" containerID="dca0d9584297e37f794a7abb2dcd6cab6829b6f751396b5a81d2c2ed44b534e9"
Oct 31 13:46:42.231952 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.231925    4660 generic.go:356] "Generic (PLEG): container finished" podID="fecef293-ce25-4fc8-84cf-91140172b297" containerID="66382f62e91bbbb7698cc561d8e1d3d88decadb0cf17fa2f01b1951b7fd4b331" exitCode=0
Oct 31 13:46:42.232073 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.232001    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6" event={"ID":"fecef293-ce25-4fc8-84cf-91140172b297","Type":"ContainerDied","Data":"66382f62e91bbbb7698cc561d8e1d3d88decadb0cf17fa2f01b1951b7fd4b331"}
Oct 31 13:46:42.232073 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.232052    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6" event={"ID":"fecef293-ce25-4fc8-84cf-91140172b297","Type":"ContainerDied","Data":"20e3cd5c876065ca4fb1a9b56019a2506108242398de70b011564263a671f8c8"}
Oct 31 13:46:42.232151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.232129    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"
Oct 31 13:46:42.247791 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.247775    4660 scope.go:122] "RemoveContainer" containerID="dca0d9584297e37f794a7abb2dcd6cab6829b6f751396b5a81d2c2ed44b534e9"
Oct 31 13:46:42.248049 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:46:42.248027    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca0d9584297e37f794a7abb2dcd6cab6829b6f751396b5a81d2c2ed44b534e9\": container with ID starting with dca0d9584297e37f794a7abb2dcd6cab6829b6f751396b5a81d2c2ed44b534e9 not found: ID does not exist" containerID="dca0d9584297e37f794a7abb2dcd6cab6829b6f751396b5a81d2c2ed44b534e9"
Oct 31 13:46:42.248112 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.248057    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca0d9584297e37f794a7abb2dcd6cab6829b6f751396b5a81d2c2ed44b534e9"} err="failed to get container status \"dca0d9584297e37f794a7abb2dcd6cab6829b6f751396b5a81d2c2ed44b534e9\": rpc error: code = NotFound desc = could not find container \"dca0d9584297e37f794a7abb2dcd6cab6829b6f751396b5a81d2c2ed44b534e9\": container with ID starting with dca0d9584297e37f794a7abb2dcd6cab6829b6f751396b5a81d2c2ed44b534e9 not found: ID does not exist"
Oct 31 13:46:42.248112 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.248073    4660 scope.go:122] "RemoveContainer" containerID="66382f62e91bbbb7698cc561d8e1d3d88decadb0cf17fa2f01b1951b7fd4b331"
Oct 31 13:46:42.258670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.258650    4660 scope.go:122] "RemoveContainer" containerID="66382f62e91bbbb7698cc561d8e1d3d88decadb0cf17fa2f01b1951b7fd4b331"
Oct 31 13:46:42.258968 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:46:42.258948    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66382f62e91bbbb7698cc561d8e1d3d88decadb0cf17fa2f01b1951b7fd4b331\": container with ID starting with 66382f62e91bbbb7698cc561d8e1d3d88decadb0cf17fa2f01b1951b7fd4b331 not found: ID does not exist" containerID="66382f62e91bbbb7698cc561d8e1d3d88decadb0cf17fa2f01b1951b7fd4b331"
Oct 31 13:46:42.259019 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.258976    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66382f62e91bbbb7698cc561d8e1d3d88decadb0cf17fa2f01b1951b7fd4b331"} err="failed to get container status \"66382f62e91bbbb7698cc561d8e1d3d88decadb0cf17fa2f01b1951b7fd4b331\": rpc error: code = NotFound desc = could not find container \"66382f62e91bbbb7698cc561d8e1d3d88decadb0cf17fa2f01b1951b7fd4b331\": container with ID starting with 66382f62e91bbbb7698cc561d8e1d3d88decadb0cf17fa2f01b1951b7fd4b331 not found: ID does not exist"
Oct 31 13:46:42.266568 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.266546    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"]
Oct 31 13:46:42.271136 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.271111    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-h52p6"]
Oct 31 13:46:42.284530 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.284506    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"]
Oct 31 13:46:42.295842 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.295820    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-kw4bj"]
Oct 31 13:46:42.532525 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.532448    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="239235d5-45ef-405c-8c3a-02618f5ebaa0" path="/var/lib/kubelet/pods/239235d5-45ef-405c-8c3a-02618f5ebaa0/volumes"
Oct 31 13:46:42.533194 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:42.533174    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fecef293-ce25-4fc8-84cf-91140172b297" path="/var/lib/kubelet/pods/fecef293-ce25-4fc8-84cf-91140172b297/volumes"
Oct 31 13:46:46.867329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:46.866284    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"]
Oct 31 13:46:46.867329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:46.866525    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm" podUID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" containerName="istio-proxy" containerID="cri-o://c74014eac6f8f8891971e981ce3daaa2cade9a001608317240864b7f9df65f1c" gracePeriod=30
Oct 31 13:46:48.966042 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:48.966003    4660 patch_prober.go:29] interesting pod/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:46:48.966421 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:48.966077    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm" podUID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:46:50.747356 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:46:50.747314    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:46:52.193757 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.193731    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:46:52.289998 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.289965    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-envoy\") pod \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") "
Oct 31 13:46:52.289998 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.290011    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istiod-ca-cert\") pod \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") "
Oct 31 13:46:52.290592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.290084    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-data\") pod \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") "
Oct 31 13:46:52.290592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.290110    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-token\" (UniqueName: \"kubernetes.io/projected/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-token\") pod \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") "
Oct 31 13:46:52.290592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.290147    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-podinfo\") pod \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") "
Oct 31 13:46:52.290592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.290269    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-workload-certs\") pod \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") "
Oct 31 13:46:52.290592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.290441    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a52dcc2-c2ab-430c-988e-d3b967f833f3-kube-api-access-l9s6k\" (UniqueName: \"kubernetes.io/projected/8a52dcc2-c2ab-430c-988e-d3b967f833f3-kube-api-access-l9s6k\") pod \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") "
Oct 31 13:46:52.290592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.290499    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istiod-ca-cert" pod "8a52dcc2-c2ab-430c-988e-d3b967f833f3" (UID: "8a52dcc2-c2ab-430c-988e-d3b967f833f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:52.290592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.290536    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-credential-socket\") pod \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") "
Oct 31 13:46:52.291065 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.290601    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-workload-certs" pod "8a52dcc2-c2ab-430c-988e-d3b967f833f3" (UID: "8a52dcc2-c2ab-430c-988e-d3b967f833f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:52.291065 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.290755    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-workload-socket\") pod \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\" (UID: \"8a52dcc2-c2ab-430c-988e-d3b967f833f3\") "
Oct 31 13:46:52.291065 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.290804    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-data" pod "8a52dcc2-c2ab-430c-988e-d3b967f833f3" (UID: "8a52dcc2-c2ab-430c-988e-d3b967f833f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:52.291065 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.290883    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-credential-socket" pod "8a52dcc2-c2ab-430c-988e-d3b967f833f3" (UID: "8a52dcc2-c2ab-430c-988e-d3b967f833f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:52.291065 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.291021    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-workload-socket" pod "8a52dcc2-c2ab-430c-988e-d3b967f833f3" (UID: "8a52dcc2-c2ab-430c-988e-d3b967f833f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:52.291541 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.291509    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:52.291541 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.291530    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:52.291541 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.291545    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:52.291750 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.291561    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:52.291750 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.291574    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:52.292852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.292822    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a52dcc2-c2ab-430c-988e-d3b967f833f3-kube-api-access-l9s6k" pod "8a52dcc2-c2ab-430c-988e-d3b967f833f3" (UID: "8a52dcc2-c2ab-430c-988e-d3b967f833f3"). InnerVolumeSpecName "kube-api-access-l9s6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:52.292966 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.292905    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-podinfo" pod "8a52dcc2-c2ab-430c-988e-d3b967f833f3" (UID: "8a52dcc2-c2ab-430c-988e-d3b967f833f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:46:52.293374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.293355    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-token" pod "8a52dcc2-c2ab-430c-988e-d3b967f833f3" (UID: "8a52dcc2-c2ab-430c-988e-d3b967f833f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:52.293440 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.293400    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-envoy" pod "8a52dcc2-c2ab-430c-988e-d3b967f833f3" (UID: "8a52dcc2-c2ab-430c-988e-d3b967f833f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:52.393208 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.393174    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9s6k\" (UniqueName: \"kubernetes.io/projected/8a52dcc2-c2ab-430c-988e-d3b967f833f3-kube-api-access-l9s6k\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:52.393208 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.393206    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:52.393208 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.393216    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:52.393459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.393225    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a52dcc2-c2ab-430c-988e-d3b967f833f3-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:52.529140 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.529059    4660 generic.go:356] "Generic (PLEG): container finished" podID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" containerID="c74014eac6f8f8891971e981ce3daaa2cade9a001608317240864b7f9df65f1c" exitCode=0
Oct 31 13:46:52.529325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.529275    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:46:52.535408 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.535360    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm" event={"ID":"8a52dcc2-c2ab-430c-988e-d3b967f833f3","Type":"ContainerDied","Data":"c74014eac6f8f8891971e981ce3daaa2cade9a001608317240864b7f9df65f1c"}
Oct 31 13:46:52.535556 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.535423    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm" event={"ID":"8a52dcc2-c2ab-430c-988e-d3b967f833f3","Type":"ContainerDied","Data":"4265fd12f0f114a9f689c6fce156765802fea8aec3e8223b2c60d8bf85845f6c"}
Oct 31 13:46:52.535556 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.535456    4660 scope.go:122] "RemoveContainer" containerID="c74014eac6f8f8891971e981ce3daaa2cade9a001608317240864b7f9df65f1c"
Oct 31 13:46:52.553903 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.553882    4660 scope.go:122] "RemoveContainer" containerID="e365c0fcce366ead8f5df728c37af26995d99ecaf27ad58f9b5e2ecd7c4644fc"
Oct 31 13:46:52.564789 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.564772    4660 scope.go:122] "RemoveContainer" containerID="c74014eac6f8f8891971e981ce3daaa2cade9a001608317240864b7f9df65f1c"
Oct 31 13:46:52.565074 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:46:52.565055    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c74014eac6f8f8891971e981ce3daaa2cade9a001608317240864b7f9df65f1c\": container with ID starting with c74014eac6f8f8891971e981ce3daaa2cade9a001608317240864b7f9df65f1c not found: ID does not exist" containerID="c74014eac6f8f8891971e981ce3daaa2cade9a001608317240864b7f9df65f1c"
Oct 31 13:46:52.565119 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.565084    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c74014eac6f8f8891971e981ce3daaa2cade9a001608317240864b7f9df65f1c"} err="failed to get container status \"c74014eac6f8f8891971e981ce3daaa2cade9a001608317240864b7f9df65f1c\": rpc error: code = NotFound desc = could not find container \"c74014eac6f8f8891971e981ce3daaa2cade9a001608317240864b7f9df65f1c\": container with ID starting with c74014eac6f8f8891971e981ce3daaa2cade9a001608317240864b7f9df65f1c not found: ID does not exist"
Oct 31 13:46:52.565119 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.565103    4660 scope.go:122] "RemoveContainer" containerID="e365c0fcce366ead8f5df728c37af26995d99ecaf27ad58f9b5e2ecd7c4644fc"
Oct 31 13:46:52.565302 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:46:52.565271    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e365c0fcce366ead8f5df728c37af26995d99ecaf27ad58f9b5e2ecd7c4644fc\": container with ID starting with e365c0fcce366ead8f5df728c37af26995d99ecaf27ad58f9b5e2ecd7c4644fc not found: ID does not exist" containerID="e365c0fcce366ead8f5df728c37af26995d99ecaf27ad58f9b5e2ecd7c4644fc"
Oct 31 13:46:52.565353 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:52.565307    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e365c0fcce366ead8f5df728c37af26995d99ecaf27ad58f9b5e2ecd7c4644fc"} err="failed to get container status \"e365c0fcce366ead8f5df728c37af26995d99ecaf27ad58f9b5e2ecd7c4644fc\": rpc error: code = NotFound desc = could not find container \"e365c0fcce366ead8f5df728c37af26995d99ecaf27ad58f9b5e2ecd7c4644fc\": container with ID starting with e365c0fcce366ead8f5df728c37af26995d99ecaf27ad58f9b5e2ecd7c4644fc not found: ID does not exist"
Oct 31 13:46:59.974458 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:59.974223    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"]
Oct 31 13:46:59.974957 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:46:59.974489    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h" podUID="51ef7324-3450-4432-a5ee-30c91193f7a0" containerName="maas-api" containerID="cri-o://b66941891f46556271083fc751020a3063f974ce356407c9f01e8942d1ff4c5c" gracePeriod=30
Oct 31 13:47:00.577430 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.577396    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:47:00.647325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.644925    4660 generic.go:356] "Generic (PLEG): container finished" podID="51ef7324-3450-4432-a5ee-30c91193f7a0" containerID="b66941891f46556271083fc751020a3063f974ce356407c9f01e8942d1ff4c5c" exitCode=0
Oct 31 13:47:00.647325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.645234    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h" event={"ID":"51ef7324-3450-4432-a5ee-30c91193f7a0","Type":"ContainerDied","Data":"b66941891f46556271083fc751020a3063f974ce356407c9f01e8942d1ff4c5c"}
Oct 31 13:47:00.647325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.645379    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h" event={"ID":"51ef7324-3450-4432-a5ee-30c91193f7a0","Type":"ContainerDied","Data":"afff2fc6505e13e9541b0be8bdaaa1b88a80050e2ed440ce756915b8f38b8107"}
Oct 31 13:47:00.647325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.645424    4660 scope.go:122] "RemoveContainer" containerID="b66941891f46556271083fc751020a3063f974ce356407c9f01e8942d1ff4c5c"
Oct 31 13:47:00.647325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.645599    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"
Oct 31 13:47:00.665740 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.665630    4660 scope.go:122] "RemoveContainer" containerID="b66941891f46556271083fc751020a3063f974ce356407c9f01e8942d1ff4c5c"
Oct 31 13:47:00.666692 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:47:00.666629    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b66941891f46556271083fc751020a3063f974ce356407c9f01e8942d1ff4c5c\": container with ID starting with b66941891f46556271083fc751020a3063f974ce356407c9f01e8942d1ff4c5c not found: ID does not exist" containerID="b66941891f46556271083fc751020a3063f974ce356407c9f01e8942d1ff4c5c"
Oct 31 13:47:00.666692 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.666662    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b66941891f46556271083fc751020a3063f974ce356407c9f01e8942d1ff4c5c"} err="failed to get container status \"b66941891f46556271083fc751020a3063f974ce356407c9f01e8942d1ff4c5c\": rpc error: code = NotFound desc = could not find container \"b66941891f46556271083fc751020a3063f974ce356407c9f01e8942d1ff4c5c\": container with ID starting with b66941891f46556271083fc751020a3063f974ce356407c9f01e8942d1ff4c5c not found: ID does not exist"
Oct 31 13:47:00.691406 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.691376    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ef7324-3450-4432-a5ee-30c91193f7a0-kube-api-access-ndmjj\" (UniqueName: \"kubernetes.io/projected/51ef7324-3450-4432-a5ee-30c91193f7a0-kube-api-access-ndmjj\") pod \"51ef7324-3450-4432-a5ee-30c91193f7a0\" (UID: \"51ef7324-3450-4432-a5ee-30c91193f7a0\") "
Oct 31 13:47:00.691595 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.691471    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51ef7324-3450-4432-a5ee-30c91193f7a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51ef7324-3450-4432-a5ee-30c91193f7a0-maas-api-tls\") pod \"51ef7324-3450-4432-a5ee-30c91193f7a0\" (UID: \"51ef7324-3450-4432-a5ee-30c91193f7a0\") "
Oct 31 13:47:00.691595 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.691584    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ef7324-3450-4432-a5ee-30c91193f7a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ef7324-3450-4432-a5ee-30c91193f7a0-openshift-service-ca\") pod \"51ef7324-3450-4432-a5ee-30c91193f7a0\" (UID: \"51ef7324-3450-4432-a5ee-30c91193f7a0\") "
Oct 31 13:47:00.691745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.691727    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51ef7324-3450-4432-a5ee-30c91193f7a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51ef7324-3450-4432-a5ee-30c91193f7a0-metrics-certs\") pod \"51ef7324-3450-4432-a5ee-30c91193f7a0\" (UID: \"51ef7324-3450-4432-a5ee-30c91193f7a0\") "
Oct 31 13:47:00.692066 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.692031    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ef7324-3450-4432-a5ee-30c91193f7a0-openshift-service-ca" pod "51ef7324-3450-4432-a5ee-30c91193f7a0" (UID: "51ef7324-3450-4432-a5ee-30c91193f7a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:47:00.692902 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.692875    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ef7324-3450-4432-a5ee-30c91193f7a0-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:00.693816 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.693779    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ef7324-3450-4432-a5ee-30c91193f7a0-kube-api-access-ndmjj" pod "51ef7324-3450-4432-a5ee-30c91193f7a0" (UID: "51ef7324-3450-4432-a5ee-30c91193f7a0"). InnerVolumeSpecName "kube-api-access-ndmjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:47:00.693917 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.693885    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ef7324-3450-4432-a5ee-30c91193f7a0-maas-api-tls" pod "51ef7324-3450-4432-a5ee-30c91193f7a0" (UID: "51ef7324-3450-4432-a5ee-30c91193f7a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:47:00.694236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.694212    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ef7324-3450-4432-a5ee-30c91193f7a0-metrics-certs" pod "51ef7324-3450-4432-a5ee-30c91193f7a0" (UID: "51ef7324-3450-4432-a5ee-30c91193f7a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:47:00.795382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.794443    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51ef7324-3450-4432-a5ee-30c91193f7a0-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:00.795382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.794467    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51ef7324-3450-4432-a5ee-30c91193f7a0-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:00.795382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.794478    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndmjj\" (UniqueName: \"kubernetes.io/projected/51ef7324-3450-4432-a5ee-30c91193f7a0-kube-api-access-ndmjj\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:00.980606 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.980571    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"]
Oct 31 13:47:00.989792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:00.989754    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h"]
Oct 31 13:47:02.529953 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:02.529918    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51ef7324-3450-4432-a5ee-30c91193f7a0" path="/var/lib/kubelet/pods/51ef7324-3450-4432-a5ee-30c91193f7a0/volumes"
Oct 31 13:47:03.233519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:03.233453    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d835b-5dbc69897-7dj2h" podUID="51ef7324-3450-4432-a5ee-30c91193f7a0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.137:8443/health\": context deadline exceeded"
Oct 31 13:47:13.887332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:13.887265    4660 generic.go:356] "Generic (PLEG): container finished" podID="acc4f15d-78ea-4803-9d9d-ad445710d934" containerID="10cc3753da99e65c253ce224644c71ce661f1539bda8f6970036545aa220ac2a" exitCode=0
Oct 31 13:47:13.887847 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:13.887364    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx" event={"ID":"acc4f15d-78ea-4803-9d9d-ad445710d934","Type":"ContainerDied","Data":"10cc3753da99e65c253ce224644c71ce661f1539bda8f6970036545aa220ac2a"}
Oct 31 13:47:13.887937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:13.887920    4660 scope.go:122] "RemoveContainer" containerID="10cc3753da99e65c253ce224644c71ce661f1539bda8f6970036545aa220ac2a"
Oct 31 13:47:13.890130 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:13.890096    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:47:13.890275 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:13.890168    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:47:13.890275 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:13.890209    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:47:13.996649 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:13.996586    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:47:14.897599 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:14.897560    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx" event={"ID":"acc4f15d-78ea-4803-9d9d-ad445710d934","Type":"ContainerStarted","Data":"65ee145305b21d0093440be208eb34022bbaeedc3f41c2f061360b10169e71ee"}
Oct 31 13:47:15.857347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:15.854603    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:47:15.857347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:15.855592    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 31 13:47:15.857347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:15.855647    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx" podUID="acc4f15d-78ea-4803-9d9d-ad445710d934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 31 13:47:16.826318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:16.824632    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 31 13:47:16.826318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:16.824693    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx" podUID="acc4f15d-78ea-4803-9d9d-ad445710d934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 31 13:47:17.830318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:17.829702    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 31 13:47:17.830318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:17.829778    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx" podUID="acc4f15d-78ea-4803-9d9d-ad445710d934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 31 13:47:18.831432 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:18.831394    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 31 13:47:18.832035 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:18.832001    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx" podUID="acc4f15d-78ea-4803-9d9d-ad445710d934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 31 13:47:19.829317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:19.824619    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 31 13:47:19.829317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:19.824690    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx" podUID="acc4f15d-78ea-4803-9d9d-ad445710d934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 31 13:47:20.737937 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:47:20.737878    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:47:20.830027 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:20.829994    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:47:20.830570 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:20.830536    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:47:20.831331 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:20.831281    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:47:22.547365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:22.547327    4660 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod8a52dcc2-c2ab-430c-988e-d3b967f833f3"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod8a52dcc2-c2ab-430c-988e-d3b967f833f3] : Timed out while waiting for systemd to remove kubepods-burstable-pod8a52dcc2_c2ab_430c_988e_d3b967f833f3.slice"
Oct 31 13:47:22.547365 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:47:22.547367    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod8a52dcc2-c2ab-430c-988e-d3b967f833f3] : unable to destroy cgroup paths for cgroup [kubepods burstable pod8a52dcc2-c2ab-430c-988e-d3b967f833f3] : Timed out while waiting for systemd to remove kubepods-burstable-pod8a52dcc2_c2ab_430c_988e_d3b967f833f3.slice" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm" podUID="8a52dcc2-c2ab-430c-988e-d3b967f833f3"
Oct 31 13:47:23.097602 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:23.097570    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"
Oct 31 13:47:23.128158 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:23.128124    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"]
Oct 31 13:47:23.135312 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:23.135263    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4fde10-openshift-default-85c7f6f7f7-7zmjm"]
Oct 31 13:47:24.529097 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:47:24.529065    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" path="/var/lib/kubelet/pods/8a52dcc2-c2ab-430c-988e-d3b967f833f3/volumes"
Oct 31 13:47:50.741242 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:47:50.741195    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:48:03.801493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.801455    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"]
Oct 31 13:48:03.804509 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804477    4660 cpu_manager.go:414] "containerMap: removing container" podUID="77c1be44-f679-4953-9a0d-74f8c160e29c" containerName="istio-proxy"
Oct 31 13:48:03.804509 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804509    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c1be44-f679-4953-9a0d-74f8c160e29c" containerName="istio-proxy"
Oct 31 13:48:03.804681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804536    4660 cpu_manager.go:414] "containerMap: removing container" podUID="afa08b46-4029-44a9-9c58-1ae42760deb7" containerName="cleanup"
Oct 31 13:48:03.804681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804545    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa08b46-4029-44a9-9c58-1ae42760deb7" containerName="cleanup"
Oct 31 13:48:03.804681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804561    4660 cpu_manager.go:414] "containerMap: removing container" podUID="239235d5-45ef-405c-8c3a-02618f5ebaa0" containerName="istio-proxy"
Oct 31 13:48:03.804681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804569    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="239235d5-45ef-405c-8c3a-02618f5ebaa0" containerName="istio-proxy"
Oct 31 13:48:03.804681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804582    4660 cpu_manager.go:414] "containerMap: removing container" podUID="afa08b46-4029-44a9-9c58-1ae42760deb7" containerName="cleanup"
Oct 31 13:48:03.804681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804589    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa08b46-4029-44a9-9c58-1ae42760deb7" containerName="cleanup"
Oct 31 13:48:03.804681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804604    4660 cpu_manager.go:414] "containerMap: removing container" podUID="708ebdbd-2c8c-4d47-8d03-517e149613d1" containerName="main"
Oct 31 13:48:03.804681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804611    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708ebdbd-2c8c-4d47-8d03-517e149613d1" containerName="main"
Oct 31 13:48:03.804681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804624    4660 cpu_manager.go:414] "containerMap: removing container" podUID="61588bc9-c966-4ea3-9a38-750555e43840" containerName="cleanup"
Oct 31 13:48:03.804681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804631    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61588bc9-c966-4ea3-9a38-750555e43840" containerName="cleanup"
Oct 31 13:48:03.804681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804642    4660 cpu_manager.go:414] "containerMap: removing container" podUID="94082533-524e-4201-ae61-c73877895d65" containerName="cleanup"
Oct 31 13:48:03.804681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804649    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94082533-524e-4201-ae61-c73877895d65" containerName="cleanup"
Oct 31 13:48:03.804681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804660    4660 cpu_manager.go:414] "containerMap: removing container" podUID="cdc22df5-69c4-4def-9e7d-f022057cd410" containerName="main"
Oct 31 13:48:03.804681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804668    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc22df5-69c4-4def-9e7d-f022057cd410" containerName="main"
Oct 31 13:48:03.804681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804681    4660 cpu_manager.go:414] "containerMap: removing container" podUID="a37b859d-6ef3-4e77-9459-47939791f9c6" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804688    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37b859d-6ef3-4e77-9459-47939791f9c6" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804699    4660 cpu_manager.go:414] "containerMap: removing container" podUID="fef53539-9688-4af7-8ce0-5aa0bbea4f9c" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804707    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef53539-9688-4af7-8ce0-5aa0bbea4f9c" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804716    4660 cpu_manager.go:414] "containerMap: removing container" podUID="d4e738ae-b6fd-4c4b-9413-0497e07cbc03" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804723    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e738ae-b6fd-4c4b-9413-0497e07cbc03" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804751    4660 cpu_manager.go:414] "containerMap: removing container" podUID="ad8e87df-0d34-443a-ba4a-11f647aed571" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804758    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad8e87df-0d34-443a-ba4a-11f647aed571" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804770    4660 cpu_manager.go:414] "containerMap: removing container" podUID="f3e60e1e-371d-4a37-b988-e76510d17e90" containerName="main"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804776    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e60e1e-371d-4a37-b988-e76510d17e90" containerName="main"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804810    4660 cpu_manager.go:414] "containerMap: removing container" podUID="740f8e32-74e2-4c64-8d14-e0124a90bd5c" containerName="main"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804818    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="740f8e32-74e2-4c64-8d14-e0124a90bd5c" containerName="main"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804828    4660 cpu_manager.go:414] "containerMap: removing container" podUID="94082533-524e-4201-ae61-c73877895d65" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804835    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94082533-524e-4201-ae61-c73877895d65" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804853    4660 cpu_manager.go:414] "containerMap: removing container" podUID="a37b859d-6ef3-4e77-9459-47939791f9c6" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804860    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37b859d-6ef3-4e77-9459-47939791f9c6" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804884    4660 cpu_manager.go:414] "containerMap: removing container" podUID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804891    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804903    4660 cpu_manager.go:414] "containerMap: removing container" podUID="df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804909    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804921    4660 cpu_manager.go:414] "containerMap: removing container" podUID="ad8e87df-0d34-443a-ba4a-11f647aed571" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804928    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad8e87df-0d34-443a-ba4a-11f647aed571" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804938    4660 cpu_manager.go:414] "containerMap: removing container" podUID="61588bc9-c966-4ea3-9a38-750555e43840" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804945    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61588bc9-c966-4ea3-9a38-750555e43840" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804964    4660 cpu_manager.go:414] "containerMap: removing container" podUID="94082533-524e-4201-ae61-c73877895d65" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804970    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94082533-524e-4201-ae61-c73877895d65" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804983    4660 cpu_manager.go:414] "containerMap: removing container" podUID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.804991    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805011    4660 cpu_manager.go:414] "containerMap: removing container" podUID="0e4a9639-2171-4539-abca-aa29d66bcdfb" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805018    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e4a9639-2171-4539-abca-aa29d66bcdfb" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805030    4660 cpu_manager.go:414] "containerMap: removing container" podUID="ad8e87df-0d34-443a-ba4a-11f647aed571" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805037    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad8e87df-0d34-443a-ba4a-11f647aed571" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805051    4660 cpu_manager.go:414] "containerMap: removing container" podUID="d4e738ae-b6fd-4c4b-9413-0497e07cbc03" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805058    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e738ae-b6fd-4c4b-9413-0497e07cbc03" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805072    4660 cpu_manager.go:414] "containerMap: removing container" podUID="bb79fc7d-a4bb-4e66-b320-eeff445ff5c0" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805078    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb79fc7d-a4bb-4e66-b320-eeff445ff5c0" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805089    4660 cpu_manager.go:414] "containerMap: removing container" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805097    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805108    4660 cpu_manager.go:414] "containerMap: removing container" podUID="bb79fc7d-a4bb-4e66-b320-eeff445ff5c0" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805115    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb79fc7d-a4bb-4e66-b320-eeff445ff5c0" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805126    4660 cpu_manager.go:414] "containerMap: removing container" podUID="fecef293-ce25-4fc8-84cf-91140172b297" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805133    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fecef293-ce25-4fc8-84cf-91140172b297" containerName="istio-proxy"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805142    4660 cpu_manager.go:414] "containerMap: removing container" podUID="a37b859d-6ef3-4e77-9459-47939791f9c6" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805148    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37b859d-6ef3-4e77-9459-47939791f9c6" containerName="cleanup"
Oct 31 13:48:03.805145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805159    4660 cpu_manager.go:414] "containerMap: removing container" podUID="bb79fc7d-a4bb-4e66-b320-eeff445ff5c0" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805166    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb79fc7d-a4bb-4e66-b320-eeff445ff5c0" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805181    4660 cpu_manager.go:414] "containerMap: removing container" podUID="19742f9a-ed37-4745-a0ac-af1d842a334d" containerName="maas-api"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805188    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19742f9a-ed37-4745-a0ac-af1d842a334d" containerName="maas-api"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805199    4660 cpu_manager.go:414] "containerMap: removing container" podUID="51ef7324-3450-4432-a5ee-30c91193f7a0" containerName="maas-api"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805206    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ef7324-3450-4432-a5ee-30c91193f7a0" containerName="maas-api"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805406    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad8e87df-0d34-443a-ba4a-11f647aed571" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805423    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e60e1e-371d-4a37-b988-e76510d17e90" containerName="main"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805431    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="61588bc9-c966-4ea3-9a38-750555e43840" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805442    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="740f8e32-74e2-4c64-8d14-e0124a90bd5c" containerName="main"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805449    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="94082533-524e-4201-ae61-c73877895d65" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805458    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37b859d-6ef3-4e77-9459-47939791f9c6" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805468    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" containerName="istio-proxy"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805478    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="df482cd7-2dd2-4cb0-b791-e4a9f8c1e097" containerName="istio-proxy"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805487    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad8e87df-0d34-443a-ba4a-11f647aed571" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805495    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="61588bc9-c966-4ea3-9a38-750555e43840" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805504    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="94082533-524e-4201-ae61-c73877895d65" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805511    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a52dcc2-c2ab-430c-988e-d3b967f833f3" containerName="istio-proxy"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805520    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa08b46-4029-44a9-9c58-1ae42760deb7" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805529    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e4a9639-2171-4539-abca-aa29d66bcdfb" containerName="istio-proxy"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805537    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad8e87df-0d34-443a-ba4a-11f647aed571" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805547    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e738ae-b6fd-4c4b-9413-0497e07cbc03" containerName="istio-proxy"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805556    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb79fc7d-a4bb-4e66-b320-eeff445ff5c0" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805566    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0354dc-4369-44e8-a995-5018598ea92a" containerName="istio-proxy"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805575    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb79fc7d-a4bb-4e66-b320-eeff445ff5c0" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805587    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="fecef293-ce25-4fc8-84cf-91140172b297" containerName="istio-proxy"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805597    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37b859d-6ef3-4e77-9459-47939791f9c6" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805606    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb79fc7d-a4bb-4e66-b320-eeff445ff5c0" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805618    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="19742f9a-ed37-4745-a0ac-af1d842a334d" containerName="maas-api"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805627    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ef7324-3450-4432-a5ee-30c91193f7a0" containerName="maas-api"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805636    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c1be44-f679-4953-9a0d-74f8c160e29c" containerName="istio-proxy"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805647    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa08b46-4029-44a9-9c58-1ae42760deb7" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805655    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="239235d5-45ef-405c-8c3a-02618f5ebaa0" containerName="istio-proxy"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805665    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="708ebdbd-2c8c-4d47-8d03-517e149613d1" containerName="main"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805674    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="94082533-524e-4201-ae61-c73877895d65" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805682    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc22df5-69c4-4def-9e7d-f022057cd410" containerName="main"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805692    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37b859d-6ef3-4e77-9459-47939791f9c6" containerName="cleanup"
Oct 31 13:48:03.806539 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.805703    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef53539-9688-4af7-8ce0-5aa0bbea4f9c" containerName="istio-proxy"
Oct 31 13:48:03.810344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.810320    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:03.816519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.816440    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-599291-serving-cert\""
Oct 31 13:48:03.829568 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.829538    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"]
Oct 31 13:48:03.908880 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.908840    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:03.909077 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.908962    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:03.909162 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.909129    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:03.909224 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:03.909172    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzxhk\" (UniqueName: \"kubernetes.io/projected/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-kube-api-access-fzxhk\") pod \"maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:04.010012 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:04.009979    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:04.010219 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:04.010199    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:04.010376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:04.010270    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzxhk\" (UniqueName: \"kubernetes.io/projected/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-kube-api-access-fzxhk\") pod \"maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:04.010376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:04.010367    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:04.011972 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:04.011851    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:04.012480 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:48:04.012446    4660 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-serving-cert: secret "maas-api-e2e-shared-b-w1-599291-serving-cert" not found
Oct 31 13:48:04.013149 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:48:04.013135    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-maas-api-tls podName:e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9 nodeName:}" failed. No retries permitted until 2026-10-31 13:48:04.513112738 +0000 UTC m=+2684.131771882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-maas-api-tls") pod "maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw" (UID: "e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9") : secret "maas-api-e2e-shared-b-w1-599291-serving-cert" not found
Oct 31 13:48:04.014859 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:04.014811    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:04.032858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:04.032778    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzxhk\" (UniqueName: \"kubernetes.io/projected/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-kube-api-access-fzxhk\") pod \"maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:04.519754 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:04.519711    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:04.524891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:04.524863    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:04.756313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:04.755665    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:05.103859 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:05.103820    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"]
Oct 31 13:48:05.750264 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:05.750230    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw" event={"ID":"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9","Type":"ContainerStarted","Data":"5a8419c27d2d81bfa71355c4fe7cc4a2c018f88832703d4f4e23c77992bf30e4"}
Oct 31 13:48:06.788837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:06.788779    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw" event={"ID":"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9","Type":"ContainerStarted","Data":"b5f9ca46310a1147437a4c1ae1f6f1eadccc8356d9020450945153c94420280e"}
Oct 31 13:48:06.789907 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:06.789883    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:06.822615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:06.822563    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw" podStartSLOduration=3.215772802 podStartE2EDuration="3.822548374s" totalImagesPullingTime="606.775572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:06.818649835 +0000 UTC m=+2686.437309002" watchObservedRunningTime="2026-10-31 13:48:06.822548374 +0000 UTC m=+2686.441207537"
Oct 31 13:48:11.038507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.038346    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"]
Oct 31 13:48:11.039506 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.039482    4660 cpu_manager.go:414] "containerMap: removing container" podUID="61588bc9-c966-4ea3-9a38-750555e43840" containerName="cleanup"
Oct 31 13:48:11.039506 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.039506    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61588bc9-c966-4ea3-9a38-750555e43840" containerName="cleanup"
Oct 31 13:48:11.039669 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.039558    4660 cpu_manager.go:414] "containerMap: removing container" podUID="afa08b46-4029-44a9-9c58-1ae42760deb7" containerName="cleanup"
Oct 31 13:48:11.039669 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.039567    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa08b46-4029-44a9-9c58-1ae42760deb7" containerName="cleanup"
Oct 31 13:48:11.039839 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.039717    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="61588bc9-c966-4ea3-9a38-750555e43840" containerName="cleanup"
Oct 31 13:48:11.039839 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.039733    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e738ae-b6fd-4c4b-9413-0497e07cbc03" containerName="istio-proxy"
Oct 31 13:48:11.039839 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.039745    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa08b46-4029-44a9-9c58-1ae42760deb7" containerName="cleanup"
Oct 31 13:48:11.044332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.044278    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.057236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.057197    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"]
Oct 31 13:48:11.066251 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.058203    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1ddade-openshift-default-dockercfg-vl7sd\""
Oct 31 13:48:11.162344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.162308    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-credential-socket\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.162491 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.162370    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-workload-certs\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.162491 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.162403    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-data\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.162491 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.162431    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea4837c-09ce-427d-9a97-49c7dd077463-istiod-ca-cert\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.162622 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.162510    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-envoy\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.162622 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.162559    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-token\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.162687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.162669    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-workload-socket\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.162721 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.162692    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfvmm\" (UniqueName: \"kubernetes.io/projected/1ea4837c-09ce-427d-9a97-49c7dd077463-kube-api-access-pfvmm\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.162721 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.162717    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-podinfo\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.264194 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.264159    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea4837c-09ce-427d-9a97-49c7dd077463-istiod-ca-cert\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.264194 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.264199    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-envoy\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.264431 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.264237    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-token\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.264431 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.264369    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-workload-socket\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.264431 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.264400    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfvmm\" (UniqueName: \"kubernetes.io/projected/1ea4837c-09ce-427d-9a97-49c7dd077463-kube-api-access-pfvmm\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.264561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.264434    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-podinfo\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.264615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.264595    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-credential-socket\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.264672 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.264659    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-workload-certs\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.264735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.264717    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-data\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.265013 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.264979    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea4837c-09ce-427d-9a97-49c7dd077463-istiod-ca-cert\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.265138 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.265056    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-credential-socket\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.265138 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.265073    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-workload-socket\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.265138 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.265087    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-data\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.265329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.265227    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-workload-certs\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.266873 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.266845    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-envoy\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.267244 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.267227    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-podinfo\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.276956 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.276932    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-token\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.277064 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.276992    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfvmm\" (UniqueName: \"kubernetes.io/projected/1ea4837c-09ce-427d-9a97-49c7dd077463-kube-api-access-pfvmm\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.384604 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.384568    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:11.729707 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.729657    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"]
Oct 31 13:48:11.730973 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:48:11.730943    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea4837c_09ce_427d_9a97_49c7dd077463.slice/crio-68e7f1447fc627d66546ccb9df11fd6b317bfaad2edc802f9125d92e3e54dab4 WatchSource:0}: Error finding container 68e7f1447fc627d66546ccb9df11fd6b317bfaad2edc802f9125d92e3e54dab4: Status 404 returned error can't find the container with id 68e7f1447fc627d66546ccb9df11fd6b317bfaad2edc802f9125d92e3e54dab4
Oct 31 13:48:11.733657 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.733622    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:48:11.733782 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.733698    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:48:11.733782 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.733734    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:48:11.847218 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.847180    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn" event={"ID":"1ea4837c-09ce-427d-9a97-49c7dd077463","Type":"ContainerStarted","Data":"0122eb68071742133ac0d6a2e59f997bf5bb1b099e9bc94c6f96b95d81d63c9f"}
Oct 31 13:48:11.847392 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.847250    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn" event={"ID":"1ea4837c-09ce-427d-9a97-49c7dd077463","Type":"ContainerStarted","Data":"68e7f1447fc627d66546ccb9df11fd6b317bfaad2edc802f9125d92e3e54dab4"}
Oct 31 13:48:11.879183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:11.879123    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn" podStartSLOduration=0.879108709 podStartE2EDuration="879.108709ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:11.874333134 +0000 UTC m=+2691.492992293" watchObservedRunningTime="2026-10-31 13:48:11.879108709 +0000 UTC m=+2691.497767872"
Oct 31 13:48:12.384774 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:12.384738    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:12.386686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:12.386660    4660 patch_prober.go:29] interesting pod/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 13:48:12.386777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:12.386717    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn" podUID="1ea4837c-09ce-427d-9a97-49c7dd077463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 13:48:13.385822 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:13.385783    4660 patch_prober.go:29] interesting pod/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 13:48:13.386240 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:13.385838    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn" podUID="1ea4837c-09ce-427d-9a97-49c7dd077463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 13:48:13.836157 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:13.836070    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 13:48:14.385777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:14.385591    4660 patch_prober.go:29] interesting pod/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 13:48:14.385777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:14.385659    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn" podUID="1ea4837c-09ce-427d-9a97-49c7dd077463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 13:48:15.385886 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:15.385850    4660 patch_prober.go:29] interesting pod/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 13:48:15.386054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:15.385913    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn" podUID="1ea4837c-09ce-427d-9a97-49c7dd077463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 13:48:16.394765 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:16.385685    4660 patch_prober.go:29] interesting pod/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 13:48:16.394765 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:16.385755    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn" podUID="1ea4837c-09ce-427d-9a97-49c7dd077463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 13:48:17.384889 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:17.384850    4660 patch_prober.go:29] interesting pod/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 13:48:17.385086 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:17.384915    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn" podUID="1ea4837c-09ce-427d-9a97-49c7dd077463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 13:48:18.402136 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:18.402095    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:18.404433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:18.404231    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:18.410328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:18.409490    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:48:20.736662 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:48:20.736620    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:48:21.128708 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:21.128676    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:48:21.137861 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:21.137834    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:48:21.143845 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:21.143803    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:48:21.150548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:21.150103    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:48:21.153552 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:21.153520    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:48:21.166254 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:21.166237    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:48:24.070317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.068428    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"]
Oct 31 13:48:24.079138 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.078278    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.079352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.079184    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"]
Oct 31 13:48:24.086364 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.084480    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-5e8f29-openshift-default-dockercfg-m5586\""
Oct 31 13:48:24.212415 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.212384    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istiod-ca-cert\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.212596 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.212468    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-workload-socket\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.212596 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.212568    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-data\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.212721 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.212682    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-envoy\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.212778 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.212738    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-podinfo\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.212911 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.212889    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-credential-socket\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.212954 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.212925    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wgl7\" (UniqueName: \"kubernetes.io/projected/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-kube-api-access-4wgl7\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.213051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.213035    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-workload-certs\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.213172 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.213156    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-token\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.317386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.315158    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-workload-certs\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.317386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.315284    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-token\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.317386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.315407    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istiod-ca-cert\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.317386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.315479    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-workload-socket\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.317386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.315499    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-data\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.317386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.315543    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-envoy\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.317386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.315567    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-podinfo\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.317386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.315626    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-credential-socket\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.317386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.315643    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wgl7\" (UniqueName: \"kubernetes.io/projected/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-kube-api-access-4wgl7\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.317386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.316176    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-workload-certs\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.317386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.316835    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istiod-ca-cert\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.317386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.316999    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-workload-socket\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.317386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.317148    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-data\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.318988 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.318960    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-credential-socket\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.319707 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.319649    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-envoy\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.322624 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.322549    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-podinfo\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.331315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.330846    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-token\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.333538 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.333075    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wgl7\" (UniqueName: \"kubernetes.io/projected/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-kube-api-access-4wgl7\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.414315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.413875    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:24.971843 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.971814    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"]
Oct 31 13:48:24.972523 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:48:24.972498    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded888973_2d46_4359_8d0e_dd2e7f6fc0b2.slice/crio-22d1185cfe69de9d657be4d35bd8d4ceb2ff91b38ff58ee211497eac61dbc4b7 WatchSource:0}: Error finding container 22d1185cfe69de9d657be4d35bd8d4ceb2ff91b38ff58ee211497eac61dbc4b7: Status 404 returned error can't find the container with id 22d1185cfe69de9d657be4d35bd8d4ceb2ff91b38ff58ee211497eac61dbc4b7
Oct 31 13:48:24.975741 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.975706    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:48:24.975855 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.975790    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:48:24.975855 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:24.975837    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:48:25.976649 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:25.976615    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv" event={"ID":"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2","Type":"ContainerStarted","Data":"d7dd1a939bdbc9174d69028994cb7d7f9344bc38eecef91d05b56901741fba03"}
Oct 31 13:48:25.977140 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:25.976694    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv" event={"ID":"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2","Type":"ContainerStarted","Data":"22d1185cfe69de9d657be4d35bd8d4ceb2ff91b38ff58ee211497eac61dbc4b7"}
Oct 31 13:48:26.017380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:26.017311    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv" podStartSLOduration=2.017275 podStartE2EDuration="2.017275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:26.007536239 +0000 UTC m=+2705.626195403" watchObservedRunningTime="2026-10-31 13:48:26.017275 +0000 UTC m=+2705.635934168"
Oct 31 13:48:26.427320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:26.420980    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:26.427320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:26.421158    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 13:48:26.427320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:26.421207    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv" podUID="ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 13:48:27.419314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:27.418680    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 13:48:27.419314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:27.418744    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv" podUID="ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 13:48:27.847044 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:27.846957    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"]
Oct 31 13:48:27.859843 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:27.859019    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"]
Oct 31 13:48:27.859843 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:27.859111    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 13:48:27.866390 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:27.863971    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-14ba5a-serving-cert\""
Oct 31 13:48:28.015499 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.015458    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dr2s\" (UniqueName: \"kubernetes.io/projected/de5ecce4-e294-4623-b83d-9ba9a5e74bca-kube-api-access-9dr2s\") pod \"maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb\" (UID: \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 13:48:28.015812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.015785    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de5ecce4-e294-4623-b83d-9ba9a5e74bca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb\" (UID: \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 13:48:28.015998 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.015981    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5ecce4-e294-4623-b83d-9ba9a5e74bca-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb\" (UID: \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 13:48:28.016158 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.016140    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5ecce4-e294-4623-b83d-9ba9a5e74bca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb\" (UID: \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 13:48:28.040999 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.040969    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"]
Oct 31 13:48:28.046349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.046320    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9eb56c2-99tnf"]
Oct 31 13:48:28.126315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.125369    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de5ecce4-e294-4623-b83d-9ba9a5e74bca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb\" (UID: \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 13:48:28.126315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.125479    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5ecce4-e294-4623-b83d-9ba9a5e74bca-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb\" (UID: \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 13:48:28.126315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.125540    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5ecce4-e294-4623-b83d-9ba9a5e74bca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb\" (UID: \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 13:48:28.126315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.125831    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dr2s\" (UniqueName: \"kubernetes.io/projected/de5ecce4-e294-4623-b83d-9ba9a5e74bca-kube-api-access-9dr2s\") pod \"maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb\" (UID: \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 13:48:28.130311 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.127405    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5ecce4-e294-4623-b83d-9ba9a5e74bca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb\" (UID: \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 13:48:28.134179 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.134159    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de5ecce4-e294-4623-b83d-9ba9a5e74bca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb\" (UID: \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 13:48:28.136424 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.136393    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5ecce4-e294-4623-b83d-9ba9a5e74bca-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb\" (UID: \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 13:48:28.145696 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.145674    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dr2s\" (UniqueName: \"kubernetes.io/projected/de5ecce4-e294-4623-b83d-9ba9a5e74bca-kube-api-access-9dr2s\") pod \"maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb\" (UID: \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 13:48:28.178748 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.178722    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 13:48:28.415694 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.415161    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 13:48:28.415694 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.415224    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv" podUID="ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 13:48:28.826379 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.797280    4660 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:48:28.884025 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.883397    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2bc921-7c70-4c8d-8031-ca5462e0c2dc" path="/var/lib/kubelet/pods/ec2bc921-7c70-4c8d-8031-ca5462e0c2dc/volumes"
Oct 31 13:48:28.884025 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:28.883876    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"]
Oct 31 13:48:29.029318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:29.028322    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb" event={"ID":"de5ecce4-e294-4623-b83d-9ba9a5e74bca","Type":"ContainerStarted","Data":"6424d1935cf8a100c4a64ef226ca2353195dc806b510b5dd2c3cb732588eb3d1"}
Oct 31 13:48:29.421145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:29.420706    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 13:48:29.421145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:29.420769    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv" podUID="ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 13:48:30.113458 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:30.112462    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb" event={"ID":"de5ecce4-e294-4623-b83d-9ba9a5e74bca","Type":"ContainerStarted","Data":"5c03feec4ad1a7411b0718842378ae7eee8fc9d76ca7743ca0895aa990bee7d8"}
Oct 31 13:48:30.113458 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:30.112563    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 13:48:30.147145 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:30.147081    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb" podStartSLOduration=2.671397799 podStartE2EDuration="3.147062944s" totalImagesPullingTime="475.665145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:30.145170531 +0000 UTC m=+2709.763829696" watchObservedRunningTime="2026-10-31 13:48:30.147062944 +0000 UTC m=+2709.765722109"
Oct 31 13:48:30.436318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:30.435793    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 13:48:30.436318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:30.435872    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv" podUID="ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 13:48:31.414517 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:31.414466    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 13:48:31.414965 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:31.414522    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv" podUID="ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 13:48:32.421029 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:32.420996    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:32.421524 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:32.421498    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:32.422198 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:32.422175    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 13:48:36.030891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.030850    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"]
Oct 31 13:48:36.037648 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.037616    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ce5bb1-t2l9l"]
Oct 31 13:48:36.130444 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.130411    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 13:48:36.533417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.533374    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e7cd289-424c-434f-981b-d83931c4feb7" path="/var/lib/kubelet/pods/5e7cd289-424c-434f-981b-d83931c4feb7/volumes"
Oct 31 13:48:36.608945 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.608911    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"]
Oct 31 13:48:36.617317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.616554    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.625565 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.625539    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-97927b\"/\"auth-test-model-97927b-kserve-self-signed-certs\""
Oct 31 13:48:36.632439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.627601    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"]
Oct 31 13:48:36.632629 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.628247    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-97927b\"/\"openshift-service-ca.crt\""
Oct 31 13:48:36.632721 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.628328    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-97927b\"/\"kube-root-ca.crt\""
Oct 31 13:48:36.654154 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.653539    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-97927b\"/\"default-dockercfg-s9m6x\""
Oct 31 13:48:36.779452 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.779417    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-model-cache\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.779702 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.779459    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-dshm\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.779702 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.779485    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-tmp-dir\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.779702 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.779558    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgn5l\" (UniqueName: \"kubernetes.io/projected/4e91a423-32b4-4287-a323-6d737e05d397-kube-api-access-jgn5l\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.779891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.779711    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-home\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.779891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.779746    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e91a423-32b4-4287-a323-6d737e05d397-tls-certs\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.886318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.885083    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-model-cache\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.886318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.885148    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-dshm\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.886318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.885187    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-tmp-dir\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.886318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.885223    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgn5l\" (UniqueName: \"kubernetes.io/projected/4e91a423-32b4-4287-a323-6d737e05d397-kube-api-access-jgn5l\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.886318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.885359    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-home\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.886318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.885410    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e91a423-32b4-4287-a323-6d737e05d397-tls-certs\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.886845 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.886443    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-tmp-dir\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.886845 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.886700    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-model-cache\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.886964 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.886894    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-home\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.897311 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.891064    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e91a423-32b4-4287-a323-6d737e05d397-tls-certs\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.930340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.927073    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgn5l\" (UniqueName: \"kubernetes.io/projected/4e91a423-32b4-4287-a323-6d737e05d397-kube-api-access-jgn5l\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.940161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.933570    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-dshm\") pod \"auth-test-model-97927b-kserve-6554cf8c65-wkh22\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") " pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:36.949935 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:36.948568    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:37.367453 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:37.367424    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"]
Oct 31 13:48:37.368362 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:48:37.368334    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e91a423_32b4_4287_a323_6d737e05d397.slice/crio-a16b62ba05e4f7401bc352285e96b0e6548b6bb622fb209be45c1a4c9cfc6010 WatchSource:0}: Error finding container a16b62ba05e4f7401bc352285e96b0e6548b6bb622fb209be45c1a4c9cfc6010: Status 404 returned error can't find the container with id a16b62ba05e4f7401bc352285e96b0e6548b6bb622fb209be45c1a4c9cfc6010
Oct 31 13:48:38.210880 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:38.210838    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22" event={"ID":"4e91a423-32b4-4287-a323-6d737e05d397","Type":"ContainerStarted","Data":"a449582401e4782f8df3341da1947ed01b66971abbd99598ae67543af2c06051"}
Oct 31 13:48:38.211056 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:38.210896    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22" event={"ID":"4e91a423-32b4-4287-a323-6d737e05d397","Type":"ContainerStarted","Data":"a16b62ba05e4f7401bc352285e96b0e6548b6bb622fb209be45c1a4c9cfc6010"}
Oct 31 13:48:38.243750 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:38.243694    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22" podStartSLOduration=2.243681639 podStartE2EDuration="2.243681639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:38.233908785 +0000 UTC m=+2717.852567951" watchObservedRunningTime="2026-10-31 13:48:38.243681639 +0000 UTC m=+2717.862340804"
Oct 31 13:48:38.957308 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:38.957271    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:46.957247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:46.957201    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:46.970713 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:46.970687    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:47.335237 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:47.335144    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:48:50.737348 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:48:50.737269    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:48:54.209052 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.209017    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"]
Oct 31 13:48:54.215632 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.215605    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.227048 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.227016    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-a39e7e\"/\"auth-test-model-a39e7e-kserve-self-signed-certs\""
Oct 31 13:48:54.228591 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.228274    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-a39e7e\"/\"kube-root-ca.crt\""
Oct 31 13:48:54.228766 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.228354    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-a39e7e\"/\"openshift-service-ca.crt\""
Oct 31 13:48:54.228878 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.228522    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-a39e7e\"/\"default-dockercfg-scjgf\""
Oct 31 13:48:54.229198 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.229180    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"]
Oct 31 13:48:54.346261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.343825    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-model-cache\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.346261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.343890    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-dshm\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.346261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.343918    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36787aaf-9893-4940-b801-c90aa192d629-tls-certs\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.346261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.344006    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjr8h\" (UniqueName: \"kubernetes.io/projected/36787aaf-9893-4940-b801-c90aa192d629-kube-api-access-cjr8h\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.346261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.344134    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-home\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.346261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.344223    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-tmp-dir\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.460191 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.460099    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-model-cache\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.460493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.460461    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-dshm\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.460644 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.460625    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36787aaf-9893-4940-b801-c90aa192d629-tls-certs\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.460838 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.460819    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjr8h\" (UniqueName: \"kubernetes.io/projected/36787aaf-9893-4940-b801-c90aa192d629-kube-api-access-cjr8h\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.461060 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.461042    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-home\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.461247 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.461229    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-tmp-dir\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.475174 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.461996    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-tmp-dir\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.475444 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.466632    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-model-cache\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.475578 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.467906    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-home\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.494198 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.494159    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36787aaf-9893-4940-b801-c90aa192d629-tls-certs\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.514351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.508890    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-dshm\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.514351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.509683    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjr8h\" (UniqueName: \"kubernetes.io/projected/36787aaf-9893-4940-b801-c90aa192d629-kube-api-access-cjr8h\") pod \"auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:54.554265 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:54.553977    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:48:55.065222 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:55.065177    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"]
Oct 31 13:48:55.385376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:55.385330    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f" event={"ID":"36787aaf-9893-4940-b801-c90aa192d629","Type":"ContainerStarted","Data":"47e4a5575e0e9cd37aafb32b0f8ce882b3be761c59cc6bcf9a9419ceb357f3fd"}
Oct 31 13:48:55.385797 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:55.385394    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f" event={"ID":"36787aaf-9893-4940-b801-c90aa192d629","Type":"ContainerStarted","Data":"139920ff3c34dcf0a4d827104eb16ebe5a4bdff6e0afad4a63e762e7d07c824e"}
Oct 31 13:48:55.432499 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:55.432424    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f" podStartSLOduration=1.43240524 podStartE2EDuration="1.43240524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:55.428092361 +0000 UTC m=+2735.046751525" watchObservedRunningTime="2026-10-31 13:48:55.43240524 +0000 UTC m=+2735.051064409"
Oct 31 13:48:55.558319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:48:55.556210    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:49:04.556593 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:04.556506    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:49:04.569609 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:04.569581    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:49:05.505925 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:05.505891    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:49:14.827563 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.827525    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"]
Oct 31 13:49:14.832813 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.832788    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:14.840828 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.840758    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-599291\"/\"default-dockercfg-nxdgf\""
Oct 31 13:49:14.841065 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.841026    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-599291\"/\"auth-test-model-599291-kserve-self-signed-certs\""
Oct 31 13:49:14.846531 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.846500    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"]
Oct 31 13:49:14.856561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.856536    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-599291\"/\"openshift-service-ca.crt\""
Oct 31 13:49:14.856686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.856661    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-599291\"/\"kube-root-ca.crt\""
Oct 31 13:49:14.886047 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.886009    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-home\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:14.886232 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.886080    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-model-cache\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:14.886232 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.886133    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-tmp-dir\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:14.886393 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.886357    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b621638-e4ff-45d1-9565-6c0a98870ace-tls-certs\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:14.886457 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.886408    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6xtn\" (UniqueName: \"kubernetes.io/projected/1b621638-e4ff-45d1-9565-6c0a98870ace-kube-api-access-d6xtn\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:14.886510 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.886496    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-dshm\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:14.991187 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.991147    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b621638-e4ff-45d1-9565-6c0a98870ace-tls-certs\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:14.991483 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.991457    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6xtn\" (UniqueName: \"kubernetes.io/projected/1b621638-e4ff-45d1-9565-6c0a98870ace-kube-api-access-d6xtn\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:14.991688 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.991670    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-dshm\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:14.992028 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.991910    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-home\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:14.992220 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.992196    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-model-cache\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:14.992393 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:14.992376    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-tmp-dir\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:15.036762 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:15.036723    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-tmp-dir\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:15.044317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:15.040928    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b621638-e4ff-45d1-9565-6c0a98870ace-tls-certs\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:15.044552 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:15.044479    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-home\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:15.044851 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:15.044602    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-model-cache\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:15.067165 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:15.067094    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-dshm\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:15.088061 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:15.087927    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6xtn\" (UniqueName: \"kubernetes.io/projected/1b621638-e4ff-45d1-9565-6c0a98870ace-kube-api-access-d6xtn\") pod \"auth-test-model-599291-kserve-687dccd89c-ppbpd\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") " pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:15.183282 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:15.183247    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:15.594227 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:15.594173    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"]
Oct 31 13:49:15.595272 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:49:15.595244    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b621638_e4ff_45d1_9565_6c0a98870ace.slice/crio-c2c2cf44ddefe4325c403a187b55d10f284a24a9dc8a00da7d75016943266572 WatchSource:0}: Error finding container c2c2cf44ddefe4325c403a187b55d10f284a24a9dc8a00da7d75016943266572: Status 404 returned error can't find the container with id c2c2cf44ddefe4325c403a187b55d10f284a24a9dc8a00da7d75016943266572
Oct 31 13:49:16.606923 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:16.606879    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd" event={"ID":"1b621638-e4ff-45d1-9565-6c0a98870ace","Type":"ContainerStarted","Data":"5f525d6b22787b72405326310e88877ebaa6712ff17bd86522e203210f7fa3a6"}
Oct 31 13:49:16.606923 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:16.606933    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd" event={"ID":"1b621638-e4ff-45d1-9565-6c0a98870ace","Type":"ContainerStarted","Data":"c2c2cf44ddefe4325c403a187b55d10f284a24a9dc8a00da7d75016943266572"}
Oct 31 13:49:16.632528 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:16.632468    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd" podStartSLOduration=2.632449573 podStartE2EDuration="2.632449573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:49:16.628001635 +0000 UTC m=+2756.246660801" watchObservedRunningTime="2026-10-31 13:49:16.632449573 +0000 UTC m=+2756.251108737"
Oct 31 13:49:17.210935 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.210901    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:17.265138 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.265087    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"]
Oct 31 13:49:17.270718 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.270692    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.275700 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.275669    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-974711\"/\"openshift-service-ca.crt\""
Oct 31 13:49:17.275831 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.275729    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-974711\"/\"kube-root-ca.crt\""
Oct 31 13:49:17.278649 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.278621    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-974711\"/\"default-dockercfg-4szqx\""
Oct 31 13:49:17.278781 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.278710    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-974711\"/\"auth-test-model-974711-kserve-self-signed-certs\""
Oct 31 13:49:17.286512 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.282088    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"]
Oct 31 13:49:17.390272 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.390235    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-tls-certs\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.390445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.390315    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqr2\" (UniqueName: \"kubernetes.io/projected/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-kube-api-access-4nqr2\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.390445 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.390364    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-dshm\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.390544 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.390449    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-home\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.390544 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.390530    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-tmp-dir\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.390614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.390564    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-model-cache\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.507377 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.507135    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-dshm\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.507377 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.507284    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-home\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.507625 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.507417    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-tmp-dir\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.507625 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.507465    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-model-cache\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.507625 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.507595    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-tls-certs\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.507785 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.507656    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqr2\" (UniqueName: \"kubernetes.io/projected/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-kube-api-access-4nqr2\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.511318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.508755    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-tmp-dir\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.511318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.509024    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-home\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.511318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.509267    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-model-cache\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.538317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.523442    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-tls-certs\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.541175 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.541143    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-dshm\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.600327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.599100    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqr2\" (UniqueName: \"kubernetes.io/projected/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-kube-api-access-4nqr2\") pod \"auth-test-model-974711-kserve-8d45f44f-hzq8r\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") " pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:17.895547 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:17.895510    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:18.200988 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:18.200960    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"]
Oct 31 13:49:18.201580 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:49:18.201548    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf682ebee_c7de_42ae_a1d1_3242a0a2f0e7.slice/crio-55c7f67264ef4be8da81435547ce14e300dfa3e91f60160336099cfdd64990b1 WatchSource:0}: Error finding container 55c7f67264ef4be8da81435547ce14e300dfa3e91f60160336099cfdd64990b1: Status 404 returned error can't find the container with id 55c7f67264ef4be8da81435547ce14e300dfa3e91f60160336099cfdd64990b1
Oct 31 13:49:18.654818 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:18.654778    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r" event={"ID":"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7","Type":"ContainerStarted","Data":"c8d456b3288f73ecdf12f8eabf05e617bb0254ac7dac81ecfed42237fa870e9d"}
Oct 31 13:49:18.654977 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:18.654831    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r" event={"ID":"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7","Type":"ContainerStarted","Data":"55c7f67264ef4be8da81435547ce14e300dfa3e91f60160336099cfdd64990b1"}
Oct 31 13:49:18.677127 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:18.677069    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r" podStartSLOduration=1.677052471 podStartE2EDuration="1.677052471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:49:18.676423575 +0000 UTC m=+2758.295082740" watchObservedRunningTime="2026-10-31 13:49:18.677052471 +0000 UTC m=+2758.295711636"
Oct 31 13:49:18.896196 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:18.896164    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:20.736170 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:49:20.736130    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:49:24.768911 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:24.768870    4660 scope.go:122] "RemoveContainer" containerID="061f26d8b70298ce48c23c493c48dec24510427fedecdc0c987c5fda2ae1a567"
Oct 31 13:49:25.211270 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:25.211236    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:25.224796 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:25.224751    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:25.734127 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:25.734098    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:49:27.896566 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:27.896520    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:27.909428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:27.909404    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:28.757088 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:28.757055    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:49:33.795198 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:33.795164    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"]
Oct 31 13:49:34.848921 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:34.848876    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"]
Oct 31 13:49:46.328316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:46.327718    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"]
Oct 31 13:49:48.876286 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:48.876243    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584" podUID="59caaca5-1f5a-42fc-841d-234c76fef93e" containerName="main" containerID="cri-o://7f06008b50b7f2e9cdbbd016cc5b477e7aca71d4080e09ab3953545ad698b42f" gracePeriod=45
Oct 31 13:49:49.007102 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.007049    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584" podUID="59caaca5-1f5a-42fc-841d-234c76fef93e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.147:8000/ready\": dial tcp 10.133.0.147:8000: connect: connection refused"
Oct 31 13:49:49.009897 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.009869    4660 generic.go:356] "Generic (PLEG): container finished" podID="59caaca5-1f5a-42fc-841d-234c76fef93e" containerID="7f06008b50b7f2e9cdbbd016cc5b477e7aca71d4080e09ab3953545ad698b42f" exitCode=0
Oct 31 13:49:49.010021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.009957    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584" event={"ID":"59caaca5-1f5a-42fc-841d-234c76fef93e","Type":"ContainerDied","Data":"7f06008b50b7f2e9cdbbd016cc5b477e7aca71d4080e09ab3953545ad698b42f"}
Oct 31 13:49:49.242724 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.242694    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:49:49.353738 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.353701    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59caaca5-1f5a-42fc-841d-234c76fef93e-kube-api-access-vfft5\" (UniqueName: \"kubernetes.io/projected/59caaca5-1f5a-42fc-841d-234c76fef93e-kube-api-access-vfft5\") pod \"59caaca5-1f5a-42fc-841d-234c76fef93e\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") "
Oct 31 13:49:49.353948 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.353805    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-dshm\") pod \"59caaca5-1f5a-42fc-841d-234c76fef93e\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") "
Oct 31 13:49:49.353948 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.353830    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-home\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-home\") pod \"59caaca5-1f5a-42fc-841d-234c76fef93e\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") "
Oct 31 13:49:49.353948 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.353890    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-model-cache\") pod \"59caaca5-1f5a-42fc-841d-234c76fef93e\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") "
Oct 31 13:49:49.355259 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.353963    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59caaca5-1f5a-42fc-841d-234c76fef93e-tls-certs\" (UniqueName: \"kubernetes.io/secret/59caaca5-1f5a-42fc-841d-234c76fef93e-tls-certs\") pod \"59caaca5-1f5a-42fc-841d-234c76fef93e\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") "
Oct 31 13:49:49.355259 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.353978    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-tmp-dir\") pod \"59caaca5-1f5a-42fc-841d-234c76fef93e\" (UID: \"59caaca5-1f5a-42fc-841d-234c76fef93e\") "
Oct 31 13:49:49.355259 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.354666    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-tmp-dir" pod "59caaca5-1f5a-42fc-841d-234c76fef93e" (UID: "59caaca5-1f5a-42fc-841d-234c76fef93e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:49:49.355259 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.355185    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-home" pod "59caaca5-1f5a-42fc-841d-234c76fef93e" (UID: "59caaca5-1f5a-42fc-841d-234c76fef93e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:49:49.355259 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.355200    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-model-cache" pod "59caaca5-1f5a-42fc-841d-234c76fef93e" (UID: "59caaca5-1f5a-42fc-841d-234c76fef93e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:49:49.357723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.357406    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59caaca5-1f5a-42fc-841d-234c76fef93e-kube-api-access-vfft5" pod "59caaca5-1f5a-42fc-841d-234c76fef93e" (UID: "59caaca5-1f5a-42fc-841d-234c76fef93e"). InnerVolumeSpecName "kube-api-access-vfft5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:49:49.357723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.358273    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59caaca5-1f5a-42fc-841d-234c76fef93e-tls-certs" pod "59caaca5-1f5a-42fc-841d-234c76fef93e" (UID: "59caaca5-1f5a-42fc-841d-234c76fef93e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:49:49.357723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.358271    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-dshm" pod "59caaca5-1f5a-42fc-841d-234c76fef93e" (UID: "59caaca5-1f5a-42fc-841d-234c76fef93e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:49:49.456355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.456243    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfft5\" (UniqueName: \"kubernetes.io/projected/59caaca5-1f5a-42fc-841d-234c76fef93e-kube-api-access-vfft5\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:49:49.456355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.456280    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:49:49.456355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.456303    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:49:49.456355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.456318    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:49:49.456355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.456331    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59caaca5-1f5a-42fc-841d-234c76fef93e-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:49:49.456355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.456340    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59caaca5-1f5a-42fc-841d-234c76fef93e-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:49:49.746114 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.746025    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"]
Oct 31 13:49:49.899820 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:49.894851    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22" podUID="4e91a423-32b4-4287-a323-6d737e05d397" containerName="main" containerID="cri-o://a449582401e4782f8df3341da1947ed01b66971abbd99598ae67543af2c06051" gracePeriod=45
Oct 31 13:49:50.079428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.071219    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584" event={"ID":"59caaca5-1f5a-42fc-841d-234c76fef93e","Type":"ContainerDied","Data":"9eb33216af8981cc802c5c565019963db10b0270c08026496defb5d8e400526a"}
Oct 31 13:49:50.079428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.071281    4660 scope.go:122] "RemoveContainer" containerID="7f06008b50b7f2e9cdbbd016cc5b477e7aca71d4080e09ab3953545ad698b42f"
Oct 31 13:49:50.079428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.072658    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"
Oct 31 13:49:50.142031 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.141992    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"]
Oct 31 13:49:50.152550 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.152438    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-554759/test-model-554759-kserve-7c7c744997-p9584"]
Oct 31 13:49:50.403188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.402102    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:49:50.499317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.498653    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-dshm\") pod \"4e91a423-32b4-4287-a323-6d737e05d397\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") "
Oct 31 13:49:50.499317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.498783    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-model-cache\") pod \"4e91a423-32b4-4287-a323-6d737e05d397\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") "
Oct 31 13:49:50.499317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.498832    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-tmp-dir\") pod \"4e91a423-32b4-4287-a323-6d737e05d397\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") "
Oct 31 13:49:50.499317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.498884    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-home\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-home\") pod \"4e91a423-32b4-4287-a323-6d737e05d397\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") "
Oct 31 13:49:50.499317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.498930    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e91a423-32b4-4287-a323-6d737e05d397-kube-api-access-jgn5l\" (UniqueName: \"kubernetes.io/projected/4e91a423-32b4-4287-a323-6d737e05d397-kube-api-access-jgn5l\") pod \"4e91a423-32b4-4287-a323-6d737e05d397\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") "
Oct 31 13:49:50.499317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.499032    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e91a423-32b4-4287-a323-6d737e05d397-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e91a423-32b4-4287-a323-6d737e05d397-tls-certs\") pod \"4e91a423-32b4-4287-a323-6d737e05d397\" (UID: \"4e91a423-32b4-4287-a323-6d737e05d397\") "
Oct 31 13:49:50.534276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.512667    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-tmp-dir" pod "4e91a423-32b4-4287-a323-6d737e05d397" (UID: "4e91a423-32b4-4287-a323-6d737e05d397"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:49:50.534276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.513395    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-model-cache" pod "4e91a423-32b4-4287-a323-6d737e05d397" (UID: "4e91a423-32b4-4287-a323-6d737e05d397"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:49:50.534276 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.513650    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-home" pod "4e91a423-32b4-4287-a323-6d737e05d397" (UID: "4e91a423-32b4-4287-a323-6d737e05d397"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:49:50.554413 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.549740    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-dshm" pod "4e91a423-32b4-4287-a323-6d737e05d397" (UID: "4e91a423-32b4-4287-a323-6d737e05d397"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:49:50.612058 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.611972    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:49:50.612327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.612284    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:49:50.612459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.612444    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:49:50.612567 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.612555    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e91a423-32b4-4287-a323-6d737e05d397-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:49:50.682376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.682328    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59caaca5-1f5a-42fc-841d-234c76fef93e" path="/var/lib/kubelet/pods/59caaca5-1f5a-42fc-841d-234c76fef93e/volumes"
Oct 31 13:49:50.689111 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.689072    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e91a423-32b4-4287-a323-6d737e05d397-tls-certs" pod "4e91a423-32b4-4287-a323-6d737e05d397" (UID: "4e91a423-32b4-4287-a323-6d737e05d397"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:49:50.730328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.724441    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e91a423-32b4-4287-a323-6d737e05d397-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:49:50.730697 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.730667    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e91a423-32b4-4287-a323-6d737e05d397-kube-api-access-jgn5l" pod "4e91a423-32b4-4287-a323-6d737e05d397" (UID: "4e91a423-32b4-4287-a323-6d737e05d397"). InnerVolumeSpecName "kube-api-access-jgn5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:49:50.793360 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:49:50.793183    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:49:50.826321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:50.825844    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgn5l\" (UniqueName: \"kubernetes.io/projected/4e91a423-32b4-4287-a323-6d737e05d397-kube-api-access-jgn5l\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:49:51.162315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:51.154422    4660 generic.go:356] "Generic (PLEG): container finished" podID="4e91a423-32b4-4287-a323-6d737e05d397" containerID="a449582401e4782f8df3341da1947ed01b66971abbd99598ae67543af2c06051" exitCode=0
Oct 31 13:49:51.162315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:51.154566    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22" event={"ID":"4e91a423-32b4-4287-a323-6d737e05d397","Type":"ContainerDied","Data":"a449582401e4782f8df3341da1947ed01b66971abbd99598ae67543af2c06051"}
Oct 31 13:49:51.162315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:51.154610    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22" event={"ID":"4e91a423-32b4-4287-a323-6d737e05d397","Type":"ContainerDied","Data":"a16b62ba05e4f7401bc352285e96b0e6548b6bb622fb209be45c1a4c9cfc6010"}
Oct 31 13:49:51.162315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:51.154634    4660 scope.go:122] "RemoveContainer" containerID="a449582401e4782f8df3341da1947ed01b66971abbd99598ae67543af2c06051"
Oct 31 13:49:51.162315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:51.154704    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"
Oct 31 13:49:51.291344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:51.283519    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"]
Oct 31 13:49:51.291344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:51.288456    4660 scope.go:122] "RemoveContainer" containerID="a449582401e4782f8df3341da1947ed01b66971abbd99598ae67543af2c06051"
Oct 31 13:49:51.291344 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:49:51.288873    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a449582401e4782f8df3341da1947ed01b66971abbd99598ae67543af2c06051\": container with ID starting with a449582401e4782f8df3341da1947ed01b66971abbd99598ae67543af2c06051 not found: ID does not exist" containerID="a449582401e4782f8df3341da1947ed01b66971abbd99598ae67543af2c06051"
Oct 31 13:49:51.291344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:51.288910    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a449582401e4782f8df3341da1947ed01b66971abbd99598ae67543af2c06051"} err="failed to get container status \"a449582401e4782f8df3341da1947ed01b66971abbd99598ae67543af2c06051\": rpc error: code = NotFound desc = could not find container \"a449582401e4782f8df3341da1947ed01b66971abbd99598ae67543af2c06051\": container with ID starting with a449582401e4782f8df3341da1947ed01b66971abbd99598ae67543af2c06051 not found: ID does not exist"
Oct 31 13:49:51.304837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:51.304800    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22"]
Oct 31 13:49:51.340477 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:51.339259    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-97927b/auth-test-model-97927b-kserve-6554cf8c65-wkh22" podUID="4e91a423-32b4-4287-a323-6d737e05d397" containerName="main" probeResult="failure" output="Get \"https://10.133.0.165:8000/ready\": context deadline exceeded"
Oct 31 13:49:52.530506 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:49:52.530466    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e91a423-32b4-4287-a323-6d737e05d397" path="/var/lib/kubelet/pods/4e91a423-32b4-4287-a323-6d737e05d397/volumes"
Oct 31 13:50:01.455104 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:01.455043    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd" podUID="1b621638-e4ff-45d1-9565-6c0a98870ace" containerName="main" containerID="cri-o://5f525d6b22787b72405326310e88877ebaa6712ff17bd86522e203210f7fa3a6" gracePeriod=45
Oct 31 13:50:01.771623 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:01.771591    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:50:01.935314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:01.934179    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-tmp-dir\") pod \"1b621638-e4ff-45d1-9565-6c0a98870ace\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") "
Oct 31 13:50:01.935314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:01.934236    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b621638-e4ff-45d1-9565-6c0a98870ace-kube-api-access-d6xtn\" (UniqueName: \"kubernetes.io/projected/1b621638-e4ff-45d1-9565-6c0a98870ace-kube-api-access-d6xtn\") pod \"1b621638-e4ff-45d1-9565-6c0a98870ace\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") "
Oct 31 13:50:01.935314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:01.934355    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b621638-e4ff-45d1-9565-6c0a98870ace-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b621638-e4ff-45d1-9565-6c0a98870ace-tls-certs\") pod \"1b621638-e4ff-45d1-9565-6c0a98870ace\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") "
Oct 31 13:50:01.935314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:01.934576    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-model-cache\") pod \"1b621638-e4ff-45d1-9565-6c0a98870ace\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") "
Oct 31 13:50:01.935314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:01.934612    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-home\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-home\") pod \"1b621638-e4ff-45d1-9565-6c0a98870ace\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") "
Oct 31 13:50:01.935314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:01.934647    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-dshm\") pod \"1b621638-e4ff-45d1-9565-6c0a98870ace\" (UID: \"1b621638-e4ff-45d1-9565-6c0a98870ace\") "
Oct 31 13:50:01.940163 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:01.937566    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-tmp-dir" pod "1b621638-e4ff-45d1-9565-6c0a98870ace" (UID: "1b621638-e4ff-45d1-9565-6c0a98870ace"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:01.940163 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:01.937829    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-model-cache" pod "1b621638-e4ff-45d1-9565-6c0a98870ace" (UID: "1b621638-e4ff-45d1-9565-6c0a98870ace"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:01.940163 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:01.938011    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-home" pod "1b621638-e4ff-45d1-9565-6c0a98870ace" (UID: "1b621638-e4ff-45d1-9565-6c0a98870ace"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:01.947885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:01.943685    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-dshm" pod "1b621638-e4ff-45d1-9565-6c0a98870ace" (UID: "1b621638-e4ff-45d1-9565-6c0a98870ace"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:01.947885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:01.943977    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b621638-e4ff-45d1-9565-6c0a98870ace-tls-certs" pod "1b621638-e4ff-45d1-9565-6c0a98870ace" (UID: "1b621638-e4ff-45d1-9565-6c0a98870ace"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:50:01.947885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:01.946649    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b621638-e4ff-45d1-9565-6c0a98870ace-kube-api-access-d6xtn" pod "1b621638-e4ff-45d1-9565-6c0a98870ace" (UID: "1b621638-e4ff-45d1-9565-6c0a98870ace"). InnerVolumeSpecName "kube-api-access-d6xtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:50:02.038174 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.038070    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:02.038174 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.038114    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:02.038174 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.038129    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:02.038174 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.038142    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b621638-e4ff-45d1-9565-6c0a98870ace-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:02.038174 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.038156    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6xtn\" (UniqueName: \"kubernetes.io/projected/1b621638-e4ff-45d1-9565-6c0a98870ace-kube-api-access-d6xtn\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:02.038174 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.038170    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b621638-e4ff-45d1-9565-6c0a98870ace-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:02.286828 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.286794    4660 generic.go:356] "Generic (PLEG): container finished" podID="1b621638-e4ff-45d1-9565-6c0a98870ace" containerID="5f525d6b22787b72405326310e88877ebaa6712ff17bd86522e203210f7fa3a6" exitCode=0
Oct 31 13:50:02.287000 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.286848    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"
Oct 31 13:50:02.287000 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.286877    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd" event={"ID":"1b621638-e4ff-45d1-9565-6c0a98870ace","Type":"ContainerDied","Data":"5f525d6b22787b72405326310e88877ebaa6712ff17bd86522e203210f7fa3a6"}
Oct 31 13:50:02.287000 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.286931    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd" event={"ID":"1b621638-e4ff-45d1-9565-6c0a98870ace","Type":"ContainerDied","Data":"c2c2cf44ddefe4325c403a187b55d10f284a24a9dc8a00da7d75016943266572"}
Oct 31 13:50:02.287000 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.286954    4660 scope.go:122] "RemoveContainer" containerID="5f525d6b22787b72405326310e88877ebaa6712ff17bd86522e203210f7fa3a6"
Oct 31 13:50:02.298463 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.298438    4660 scope.go:122] "RemoveContainer" containerID="5f525d6b22787b72405326310e88877ebaa6712ff17bd86522e203210f7fa3a6"
Oct 31 13:50:02.298731 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:50:02.298712    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f525d6b22787b72405326310e88877ebaa6712ff17bd86522e203210f7fa3a6\": container with ID starting with 5f525d6b22787b72405326310e88877ebaa6712ff17bd86522e203210f7fa3a6 not found: ID does not exist" containerID="5f525d6b22787b72405326310e88877ebaa6712ff17bd86522e203210f7fa3a6"
Oct 31 13:50:02.298800 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.298742    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f525d6b22787b72405326310e88877ebaa6712ff17bd86522e203210f7fa3a6"} err="failed to get container status \"5f525d6b22787b72405326310e88877ebaa6712ff17bd86522e203210f7fa3a6\": rpc error: code = NotFound desc = could not find container \"5f525d6b22787b72405326310e88877ebaa6712ff17bd86522e203210f7fa3a6\": container with ID starting with 5f525d6b22787b72405326310e88877ebaa6712ff17bd86522e203210f7fa3a6 not found: ID does not exist"
Oct 31 13:50:02.318315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.316394    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"]
Oct 31 13:50:02.321340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.321277    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd"]
Oct 31 13:50:02.534673 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.534635    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b621638-e4ff-45d1-9565-6c0a98870ace" path="/var/lib/kubelet/pods/1b621638-e4ff-45d1-9565-6c0a98870ace/volumes"
Oct 31 13:50:02.716661 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:02.716612    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-599291/auth-test-model-599291-kserve-687dccd89c-ppbpd" podUID="1b621638-e4ff-45d1-9565-6c0a98870ace" containerName="main" probeResult="failure" output="Get \"https://10.133.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:50:03.354151 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.354116    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"]
Oct 31 13:50:03.354834 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.354813    4660 cpu_manager.go:414] "containerMap: removing container" podUID="59caaca5-1f5a-42fc-841d-234c76fef93e" containerName="main"
Oct 31 13:50:03.354834 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.354831    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59caaca5-1f5a-42fc-841d-234c76fef93e" containerName="main"
Oct 31 13:50:03.355023 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.354846    4660 cpu_manager.go:414] "containerMap: removing container" podUID="1b621638-e4ff-45d1-9565-6c0a98870ace" containerName="main"
Oct 31 13:50:03.355023 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.354851    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b621638-e4ff-45d1-9565-6c0a98870ace" containerName="main"
Oct 31 13:50:03.355023 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.354879    4660 cpu_manager.go:414] "containerMap: removing container" podUID="4e91a423-32b4-4287-a323-6d737e05d397" containerName="main"
Oct 31 13:50:03.355023 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.354884    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e91a423-32b4-4287-a323-6d737e05d397" containerName="main"
Oct 31 13:50:03.355023 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.355005    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e91a423-32b4-4287-a323-6d737e05d397" containerName="main"
Oct 31 13:50:03.355023 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.355018    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="59caaca5-1f5a-42fc-841d-234c76fef93e" containerName="main"
Oct 31 13:50:03.355023 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.355025    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b621638-e4ff-45d1-9565-6c0a98870ace" containerName="main"
Oct 31 13:50:03.363856 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.363820    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.369936 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.369911    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-97927b\"/\"kube-root-ca.crt\""
Oct 31 13:50:03.370591 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.370565    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-97927b\"/\"openshift-service-ca.crt\""
Oct 31 13:50:03.370748 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.370725    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-97927b\"/\"sub-test-model-97927b-kserve-self-signed-certs\""
Oct 31 13:50:03.370974 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.370955    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-97927b\"/\"default-dockercfg-s9m6x\""
Oct 31 13:50:03.372460 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.372433    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"]
Oct 31 13:50:03.462998 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.462952    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-tmp-dir\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.463171 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.463115    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtxpv\" (UniqueName: \"kubernetes.io/projected/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-kube-api-access-wtxpv\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.463236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.463166    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-dshm\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.463415 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.463383    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-home\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.463593 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.463570    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-tls-certs\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.463720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.463677    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-model-cache\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.576373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.575968    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtxpv\" (UniqueName: \"kubernetes.io/projected/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-kube-api-access-wtxpv\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.576373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.576029    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-dshm\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.576373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.576181    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-home\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.576373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.576312    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-tls-certs\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.576373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.576362    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-model-cache\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.577097 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.576544    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-tmp-dir\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.606364 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.596337    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-home\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.606364 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.596609    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-tmp-dir\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.606364 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.597144    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-model-cache\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.636919 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.628669    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-dshm\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.656313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.653198    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtxpv\" (UniqueName: \"kubernetes.io/projected/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-kube-api-access-wtxpv\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.656313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.653804    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-tls-certs\") pod \"sub-test-model-97927b-kserve-76569c7ff7-twn9m\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") " pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:03.719316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:03.701674    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:04.064521 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:04.064284    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"]
Oct 31 13:50:04.072459 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:50:04.072421    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c10f8e2_7ba7_4191_bb14_6a09b346f3a8.slice/crio-ec1d3593c02d48618bd69dd222bcebfe4a2526448f5b8e8b12df9b8acbc914ca WatchSource:0}: Error finding container ec1d3593c02d48618bd69dd222bcebfe4a2526448f5b8e8b12df9b8acbc914ca: Status 404 returned error can't find the container with id ec1d3593c02d48618bd69dd222bcebfe4a2526448f5b8e8b12df9b8acbc914ca
Oct 31 13:50:04.307651 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:04.307613    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m" event={"ID":"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8","Type":"ContainerStarted","Data":"e4017acfc3d16e26a78ff0c47cc41922aeb5c59c0a7dfb499f6620d031ebbe4f"}
Oct 31 13:50:04.307844 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:04.307662    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m" event={"ID":"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8","Type":"ContainerStarted","Data":"ec1d3593c02d48618bd69dd222bcebfe4a2526448f5b8e8b12df9b8acbc914ca"}
Oct 31 13:50:04.329269 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:04.329173    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m" podStartSLOduration=1.329161369 podStartE2EDuration="1.329161369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:50:04.328147187 +0000 UTC m=+2803.946806353" watchObservedRunningTime="2026-10-31 13:50:04.329161369 +0000 UTC m=+2803.947820532"
Oct 31 13:50:04.702471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:04.702432    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:04.770239 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:04.770194    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g" podUID="a5fe2659-7755-4acc-adcd-481f1d5fa982" containerName="main" containerID="cri-o://68d4fb0691c329e7d4142f1e4078a2061ed202264117ed48544e8bf0690e6d7e" gracePeriod=45
Oct 31 13:50:04.781143 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:04.781108    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g" podUID="a5fe2659-7755-4acc-adcd-481f1d5fa982" containerName="main" probeResult="failure" output="Get \"https://10.133.0.148:8000/ready\": dial tcp 10.133.0.148:8000: connect: connection refused"
Oct 31 13:50:05.082941 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.082910    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:50:05.098854 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.098724    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5fe2659-7755-4acc-adcd-481f1d5fa982-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fe2659-7755-4acc-adcd-481f1d5fa982-tls-certs\") pod \"a5fe2659-7755-4acc-adcd-481f1d5fa982\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") "
Oct 31 13:50:05.099021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.098875    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-model-cache\") pod \"a5fe2659-7755-4acc-adcd-481f1d5fa982\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") "
Oct 31 13:50:05.099021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.098906    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fe2659-7755-4acc-adcd-481f1d5fa982-kube-api-access-w4k8b\" (UniqueName: \"kubernetes.io/projected/a5fe2659-7755-4acc-adcd-481f1d5fa982-kube-api-access-w4k8b\") pod \"a5fe2659-7755-4acc-adcd-481f1d5fa982\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") "
Oct 31 13:50:05.099021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.098941    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-tmp-dir\") pod \"a5fe2659-7755-4acc-adcd-481f1d5fa982\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") "
Oct 31 13:50:05.099185 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.099036    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-dshm\") pod \"a5fe2659-7755-4acc-adcd-481f1d5fa982\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") "
Oct 31 13:50:05.099185 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.099126    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-model-cache" pod "a5fe2659-7755-4acc-adcd-481f1d5fa982" (UID: "a5fe2659-7755-4acc-adcd-481f1d5fa982"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:05.099185 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.099132    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-home\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-home\") pod \"a5fe2659-7755-4acc-adcd-481f1d5fa982\" (UID: \"a5fe2659-7755-4acc-adcd-481f1d5fa982\") "
Oct 31 13:50:05.099668 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.099641    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-tmp-dir" pod "a5fe2659-7755-4acc-adcd-481f1d5fa982" (UID: "a5fe2659-7755-4acc-adcd-481f1d5fa982"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:05.099889 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.099862    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-home" pod "a5fe2659-7755-4acc-adcd-481f1d5fa982" (UID: "a5fe2659-7755-4acc-adcd-481f1d5fa982"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:05.100022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.099962    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:05.100022 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.099985    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:05.103416 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.102742    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5fe2659-7755-4acc-adcd-481f1d5fa982-tls-certs" pod "a5fe2659-7755-4acc-adcd-481f1d5fa982" (UID: "a5fe2659-7755-4acc-adcd-481f1d5fa982"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:50:05.103416 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.103274    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fe2659-7755-4acc-adcd-481f1d5fa982-kube-api-access-w4k8b" pod "a5fe2659-7755-4acc-adcd-481f1d5fa982" (UID: "a5fe2659-7755-4acc-adcd-481f1d5fa982"). InnerVolumeSpecName "kube-api-access-w4k8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:50:05.103880 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.103854    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-dshm" pod "a5fe2659-7755-4acc-adcd-481f1d5fa982" (UID: "a5fe2659-7755-4acc-adcd-481f1d5fa982"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:05.201112 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.201079    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fe2659-7755-4acc-adcd-481f1d5fa982-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:05.201112 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.201109    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4k8b\" (UniqueName: \"kubernetes.io/projected/a5fe2659-7755-4acc-adcd-481f1d5fa982-kube-api-access-w4k8b\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:05.201351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.201121    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:05.201351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.201131    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fe2659-7755-4acc-adcd-481f1d5fa982-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:05.317084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.316994    4660 generic.go:356] "Generic (PLEG): container finished" podID="a5fe2659-7755-4acc-adcd-481f1d5fa982" containerID="68d4fb0691c329e7d4142f1e4078a2061ed202264117ed48544e8bf0690e6d7e" exitCode=0
Oct 31 13:50:05.317084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.317066    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"
Oct 31 13:50:05.317360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.317082    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g" event={"ID":"a5fe2659-7755-4acc-adcd-481f1d5fa982","Type":"ContainerDied","Data":"68d4fb0691c329e7d4142f1e4078a2061ed202264117ed48544e8bf0690e6d7e"}
Oct 31 13:50:05.317360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.317139    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g" event={"ID":"a5fe2659-7755-4acc-adcd-481f1d5fa982","Type":"ContainerDied","Data":"f88133e19a59bb62c142d276789c6871ed3c834cae0396098aa372bd53f641a1"}
Oct 31 13:50:05.317360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.317162    4660 scope.go:122] "RemoveContainer" containerID="68d4fb0691c329e7d4142f1e4078a2061ed202264117ed48544e8bf0690e6d7e"
Oct 31 13:50:05.335678 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.335656    4660 scope.go:122] "RemoveContainer" containerID="68d4fb0691c329e7d4142f1e4078a2061ed202264117ed48544e8bf0690e6d7e"
Oct 31 13:50:05.336012 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:50:05.335979    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d4fb0691c329e7d4142f1e4078a2061ed202264117ed48544e8bf0690e6d7e\": container with ID starting with 68d4fb0691c329e7d4142f1e4078a2061ed202264117ed48544e8bf0690e6d7e not found: ID does not exist" containerID="68d4fb0691c329e7d4142f1e4078a2061ed202264117ed48544e8bf0690e6d7e"
Oct 31 13:50:05.336095 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.336024    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d4fb0691c329e7d4142f1e4078a2061ed202264117ed48544e8bf0690e6d7e"} err="failed to get container status \"68d4fb0691c329e7d4142f1e4078a2061ed202264117ed48544e8bf0690e6d7e\": rpc error: code = NotFound desc = could not find container \"68d4fb0691c329e7d4142f1e4078a2061ed202264117ed48544e8bf0690e6d7e\": container with ID starting with 68d4fb0691c329e7d4142f1e4078a2061ed202264117ed48544e8bf0690e6d7e not found: ID does not exist"
Oct 31 13:50:05.350622 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.350568    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"]
Oct 31 13:50:05.355083 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:05.355058    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-35b584/test-model-35b584-kserve-77fb974cb5-mfj6g"]
Oct 31 13:50:06.529311 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:06.529259    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5fe2659-7755-4acc-adcd-481f1d5fa982" path="/var/lib/kubelet/pods/a5fe2659-7755-4acc-adcd-481f1d5fa982/volumes"
Oct 31 13:50:13.702600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:13.702561    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:13.730547 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:13.730515    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"]
Oct 31 13:50:13.730748 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:13.730725    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm" podUID="3201b01f-5dd7-4338-aa51-6899d3ae4e64" containerName="maas-api" containerID="cri-o://476ac7926d49521f758429523536f9357c8793356aa989c3cff5f8ad4cc68849" gracePeriod=30
Oct 31 13:50:13.742596 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:13.742568    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:14.375381 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.375350    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:50:14.419056 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.418176    4660 generic.go:356] "Generic (PLEG): container finished" podID="3201b01f-5dd7-4338-aa51-6899d3ae4e64" containerID="476ac7926d49521f758429523536f9357c8793356aa989c3cff5f8ad4cc68849" exitCode=0
Oct 31 13:50:14.419334 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.419313    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"
Oct 31 13:50:14.419699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.419675    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm" event={"ID":"3201b01f-5dd7-4338-aa51-6899d3ae4e64","Type":"ContainerDied","Data":"476ac7926d49521f758429523536f9357c8793356aa989c3cff5f8ad4cc68849"}
Oct 31 13:50:14.419785 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.419733    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm" event={"ID":"3201b01f-5dd7-4338-aa51-6899d3ae4e64","Type":"ContainerDied","Data":"adeaa95cfde7e1c8daac99bebeb0f6919cbb0878d4e595313888d03a07a6dfd3"}
Oct 31 13:50:14.419785 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.419771    4660 scope.go:122] "RemoveContainer" containerID="476ac7926d49521f758429523536f9357c8793356aa989c3cff5f8ad4cc68849"
Oct 31 13:50:14.437595 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.437574    4660 scope.go:122] "RemoveContainer" containerID="476ac7926d49521f758429523536f9357c8793356aa989c3cff5f8ad4cc68849"
Oct 31 13:50:14.438017 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.438000    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:50:14.438847 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:50:14.438823    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476ac7926d49521f758429523536f9357c8793356aa989c3cff5f8ad4cc68849\": container with ID starting with 476ac7926d49521f758429523536f9357c8793356aa989c3cff5f8ad4cc68849 not found: ID does not exist" containerID="476ac7926d49521f758429523536f9357c8793356aa989c3cff5f8ad4cc68849"
Oct 31 13:50:14.438989 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.438968    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476ac7926d49521f758429523536f9357c8793356aa989c3cff5f8ad4cc68849"} err="failed to get container status \"476ac7926d49521f758429523536f9357c8793356aa989c3cff5f8ad4cc68849\": rpc error: code = NotFound desc = could not find container \"476ac7926d49521f758429523536f9357c8793356aa989c3cff5f8ad4cc68849\": container with ID starting with 476ac7926d49521f758429523536f9357c8793356aa989c3cff5f8ad4cc68849 not found: ID does not exist"
Oct 31 13:50:14.482335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.479661    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3201b01f-5dd7-4338-aa51-6899d3ae4e64-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3201b01f-5dd7-4338-aa51-6899d3ae4e64-openshift-service-ca\") pod \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\" (UID: \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\") "
Oct 31 13:50:14.482335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.479840    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3201b01f-5dd7-4338-aa51-6899d3ae4e64-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3201b01f-5dd7-4338-aa51-6899d3ae4e64-maas-api-tls\") pod \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\" (UID: \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\") "
Oct 31 13:50:14.482335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.479986    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3201b01f-5dd7-4338-aa51-6899d3ae4e64-kube-api-access-jm4l7\" (UniqueName: \"kubernetes.io/projected/3201b01f-5dd7-4338-aa51-6899d3ae4e64-kube-api-access-jm4l7\") pod \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\" (UID: \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\") "
Oct 31 13:50:14.482335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.480042    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3201b01f-5dd7-4338-aa51-6899d3ae4e64-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3201b01f-5dd7-4338-aa51-6899d3ae4e64-metrics-certs\") pod \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\" (UID: \"3201b01f-5dd7-4338-aa51-6899d3ae4e64\") "
Oct 31 13:50:14.482799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.482767    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3201b01f-5dd7-4338-aa51-6899d3ae4e64-openshift-service-ca" pod "3201b01f-5dd7-4338-aa51-6899d3ae4e64" (UID: "3201b01f-5dd7-4338-aa51-6899d3ae4e64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:50:14.484532 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.484506    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3201b01f-5dd7-4338-aa51-6899d3ae4e64-metrics-certs" pod "3201b01f-5dd7-4338-aa51-6899d3ae4e64" (UID: "3201b01f-5dd7-4338-aa51-6899d3ae4e64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:50:14.486766 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.486735    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3201b01f-5dd7-4338-aa51-6899d3ae4e64-kube-api-access-jm4l7" pod "3201b01f-5dd7-4338-aa51-6899d3ae4e64" (UID: "3201b01f-5dd7-4338-aa51-6899d3ae4e64"). InnerVolumeSpecName "kube-api-access-jm4l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:50:14.486882 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.486797    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3201b01f-5dd7-4338-aa51-6899d3ae4e64-maas-api-tls" pod "3201b01f-5dd7-4338-aa51-6899d3ae4e64" (UID: "3201b01f-5dd7-4338-aa51-6899d3ae4e64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:50:14.584678 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.584573    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm4l7\" (UniqueName: \"kubernetes.io/projected/3201b01f-5dd7-4338-aa51-6899d3ae4e64-kube-api-access-jm4l7\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:14.584678 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.584616    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3201b01f-5dd7-4338-aa51-6899d3ae4e64-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:14.584678 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.584634    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3201b01f-5dd7-4338-aa51-6899d3ae4e64-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:14.584678 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.584647    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3201b01f-5dd7-4338-aa51-6899d3ae4e64-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:14.745384 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.745345    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"]
Oct 31 13:50:14.752037 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:14.751998    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-554759-d774c75df-8xqwm"]
Oct 31 13:50:16.532977 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:16.532938    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3201b01f-5dd7-4338-aa51-6899d3ae4e64" path="/var/lib/kubelet/pods/3201b01f-5dd7-4338-aa51-6899d3ae4e64/volumes"
Oct 31 13:50:19.077518 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.077425    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"]
Oct 31 13:50:19.078507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.078476    4660 cpu_manager.go:414] "containerMap: removing container" podUID="a5fe2659-7755-4acc-adcd-481f1d5fa982" containerName="main"
Oct 31 13:50:19.078507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.078503    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fe2659-7755-4acc-adcd-481f1d5fa982" containerName="main"
Oct 31 13:50:19.078715 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.078549    4660 cpu_manager.go:414] "containerMap: removing container" podUID="3201b01f-5dd7-4338-aa51-6899d3ae4e64" containerName="maas-api"
Oct 31 13:50:19.078715 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.078558    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3201b01f-5dd7-4338-aa51-6899d3ae4e64" containerName="maas-api"
Oct 31 13:50:19.078715 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.078710    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="3201b01f-5dd7-4338-aa51-6899d3ae4e64" containerName="maas-api"
Oct 31 13:50:19.078891 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.078736    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fe2659-7755-4acc-adcd-481f1d5fa982" containerName="main"
Oct 31 13:50:19.083178 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.083156    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.089177 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.089155    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-599291\"/\"default-dockercfg-nxdgf\""
Oct 31 13:50:19.090414 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.090391    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-599291\"/\"kube-root-ca.crt\""
Oct 31 13:50:19.090557 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.090407    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-599291\"/\"openshift-service-ca.crt\""
Oct 31 13:50:19.090882 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.090863    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"]
Oct 31 13:50:19.119250 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.119225    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-599291\"/\"sub-test-model-599291-kserve-self-signed-certs\""
Oct 31 13:50:19.158339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.155790    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-tmp-dir\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.158339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.156002    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-dshm\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.158339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.156057    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-home\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.158339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.156124    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-tls-certs\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.158339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.156157    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd4c2\" (UniqueName: \"kubernetes.io/projected/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-kube-api-access-sd4c2\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.158339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.156227    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-model-cache\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.262318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.259479    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-dshm\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.262318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.259561    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-home\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.262318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.259585    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-tls-certs\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.262318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.259629    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd4c2\" (UniqueName: \"kubernetes.io/projected/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-kube-api-access-sd4c2\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.262318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.259726    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-model-cache\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.262318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.259813    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-tmp-dir\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.262318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.260354    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-tmp-dir\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.262318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.261054    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-home\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.262318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.261658    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-model-cache\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.281437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.273403    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-tls-certs\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.281910 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.281877    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-dshm\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.298317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.291643    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd4c2\" (UniqueName: \"kubernetes.io/projected/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-kube-api-access-sd4c2\") pod \"sub-test-model-599291-kserve-7577667d79-z6vss\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") " pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.434332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.433568    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:19.750983 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:19.750953    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"]
Oct 31 13:50:19.753763 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:50:19.753726    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6bad0c4_bf7f_4f21_a957_8014c12c3aba.slice/crio-765873c725a540ddde415a8b0d2ab3566f7962254a1152ba7608d4b6bbffc349 WatchSource:0}: Error finding container 765873c725a540ddde415a8b0d2ab3566f7962254a1152ba7608d4b6bbffc349: Status 404 returned error can't find the container with id 765873c725a540ddde415a8b0d2ab3566f7962254a1152ba7608d4b6bbffc349
Oct 31 13:50:20.725314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:20.720969    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss" event={"ID":"d6bad0c4-bf7f-4f21-a957-8014c12c3aba","Type":"ContainerStarted","Data":"ece27ab1b7e0e714e5594adc01974413cd3ce9eedd28cd3299ad880f76f16db0"}
Oct 31 13:50:20.725314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:20.721020    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss" event={"ID":"d6bad0c4-bf7f-4f21-a957-8014c12c3aba","Type":"ContainerStarted","Data":"765873c725a540ddde415a8b0d2ab3566f7962254a1152ba7608d4b6bbffc349"}
Oct 31 13:50:20.750325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:20.749831    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss" podStartSLOduration=1.749816713 podStartE2EDuration="1.749816713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:50:20.742073073 +0000 UTC m=+2820.360732237" watchObservedRunningTime="2026-10-31 13:50:20.749816713 +0000 UTC m=+2820.368475877"
Oct 31 13:50:20.772454 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:50:20.772415    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:50:21.435723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:21.435683    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:21.982874 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:21.982834    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"]
Oct 31 13:50:21.983377 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:21.983087    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq" podUID="5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" containerName="istio-proxy" containerID="cri-o://5a54c535d6e375b40f60bc52e91b2a11adc24bea79905f516425402ae2e16688" gracePeriod=30
Oct 31 13:50:24.804261 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:24.804223    4660 scope.go:122] "RemoveContainer" containerID="e4d56369bddf62db87f48b507f8c18d099e1cbf263c7459d70ff093b839e4c01"
Oct 31 13:50:24.815495 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:24.815473    4660 scope.go:122] "RemoveContainer" containerID="b78b25373dda7a5943a532459b5d969b2f63f12aaf5d6495d195005a04265d58"
Oct 31 13:50:27.496491 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.496458    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:50:27.663375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.660909    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-data\") pod \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") "
Oct 31 13:50:27.663375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.661027    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-kube-api-access-fpm9d\" (UniqueName: \"kubernetes.io/projected/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-kube-api-access-fpm9d\") pod \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") "
Oct 31 13:50:27.663375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.661074    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-credential-socket\") pod \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") "
Oct 31 13:50:27.663375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.661151    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-workload-socket\") pod \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") "
Oct 31 13:50:27.663375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.661183    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-token\" (UniqueName: \"kubernetes.io/projected/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-token\") pod \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") "
Oct 31 13:50:27.663375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.661283    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-workload-certs\") pod \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") "
Oct 31 13:50:27.663375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.661391    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-envoy\") pod \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") "
Oct 31 13:50:27.663375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.661426    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istiod-ca-cert\") pod \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") "
Oct 31 13:50:27.663375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.661493    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-podinfo\") pod \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\" (UID: \"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e\") "
Oct 31 13:50:27.676320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.671721    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-data" pod "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" (UID: "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:27.705321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.698648    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istiod-ca-cert" pod "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" (UID: "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:50:27.705321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.698969    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-workload-socket" pod "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" (UID: "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:27.705321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.699181    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-credential-socket" pod "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" (UID: "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:27.705321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.700578    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-workload-certs" pod "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" (UID: "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:27.766688 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.764955    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:27.766688 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.764991    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:27.766688 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.765008    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:27.766688 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.765025    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:27.766688 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.765039    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:27.766688 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.765194    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-token" pod "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" (UID: "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:50:27.766688 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.765979    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-kube-api-access-fpm9d" pod "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" (UID: "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e"). InnerVolumeSpecName "kube-api-access-fpm9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:50:27.766688 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.766091    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-envoy" pod "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" (UID: "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:27.803010 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.802884    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-podinfo" pod "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" (UID: "5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:50:27.836776 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.836709    4660 generic.go:356] "Generic (PLEG): container finished" podID="5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" containerID="5a54c535d6e375b40f60bc52e91b2a11adc24bea79905f516425402ae2e16688" exitCode=0
Oct 31 13:50:27.836963 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.836869    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"
Oct 31 13:50:27.836963 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.836932    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq" event={"ID":"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e","Type":"ContainerDied","Data":"5a54c535d6e375b40f60bc52e91b2a11adc24bea79905f516425402ae2e16688"}
Oct 31 13:50:27.837095 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.836999    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq" event={"ID":"5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e","Type":"ContainerDied","Data":"de229071f12a6594355a225e629d43eb889234e7f6bd90bfebc99bfe502e8aeb"}
Oct 31 13:50:27.837095 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.837035    4660 scope.go:122] "RemoveContainer" containerID="5a54c535d6e375b40f60bc52e91b2a11adc24bea79905f516425402ae2e16688"
Oct 31 13:50:27.854245 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.854187    4660 scope.go:122] "RemoveContainer" containerID="59552b38c6f5b1cb11d559a1d59ba6b0dccdec5d8af4c337c73d3b98417a15c4"
Oct 31 13:50:27.869041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.866839    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:27.869041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.866870    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:27.869041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.866888    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:27.869041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.866906    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpm9d\" (UniqueName: \"kubernetes.io/projected/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e-kube-api-access-fpm9d\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:27.876809 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.876786    4660 scope.go:122] "RemoveContainer" containerID="5a54c535d6e375b40f60bc52e91b2a11adc24bea79905f516425402ae2e16688"
Oct 31 13:50:27.877183 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:50:27.877140    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a54c535d6e375b40f60bc52e91b2a11adc24bea79905f516425402ae2e16688\": container with ID starting with 5a54c535d6e375b40f60bc52e91b2a11adc24bea79905f516425402ae2e16688 not found: ID does not exist" containerID="5a54c535d6e375b40f60bc52e91b2a11adc24bea79905f516425402ae2e16688"
Oct 31 13:50:27.877275 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.877189    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a54c535d6e375b40f60bc52e91b2a11adc24bea79905f516425402ae2e16688"} err="failed to get container status \"5a54c535d6e375b40f60bc52e91b2a11adc24bea79905f516425402ae2e16688\": rpc error: code = NotFound desc = could not find container \"5a54c535d6e375b40f60bc52e91b2a11adc24bea79905f516425402ae2e16688\": container with ID starting with 5a54c535d6e375b40f60bc52e91b2a11adc24bea79905f516425402ae2e16688 not found: ID does not exist"
Oct 31 13:50:27.877275 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.877219    4660 scope.go:122] "RemoveContainer" containerID="59552b38c6f5b1cb11d559a1d59ba6b0dccdec5d8af4c337c73d3b98417a15c4"
Oct 31 13:50:27.877697 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:50:27.877671    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59552b38c6f5b1cb11d559a1d59ba6b0dccdec5d8af4c337c73d3b98417a15c4\": container with ID starting with 59552b38c6f5b1cb11d559a1d59ba6b0dccdec5d8af4c337c73d3b98417a15c4 not found: ID does not exist" containerID="59552b38c6f5b1cb11d559a1d59ba6b0dccdec5d8af4c337c73d3b98417a15c4"
Oct 31 13:50:27.877805 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.877707    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59552b38c6f5b1cb11d559a1d59ba6b0dccdec5d8af4c337c73d3b98417a15c4"} err="failed to get container status \"59552b38c6f5b1cb11d559a1d59ba6b0dccdec5d8af4c337c73d3b98417a15c4\": rpc error: code = NotFound desc = could not find container \"59552b38c6f5b1cb11d559a1d59ba6b0dccdec5d8af4c337c73d3b98417a15c4\": container with ID starting with 59552b38c6f5b1cb11d559a1d59ba6b0dccdec5d8af4c337c73d3b98417a15c4 not found: ID does not exist"
Oct 31 13:50:27.880656 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.880633    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"]
Oct 31 13:50:27.885936 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:27.885908    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-554759-openshift-default-69c6b6cbdf-gzhkq"]
Oct 31 13:50:28.540317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:28.540253    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" path="/var/lib/kubelet/pods/5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e/volumes"
Oct 31 13:50:29.435633 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:29.435600    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:29.452191 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:29.452158    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:29.938961 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:29.938928    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:50:50.758984 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:50:50.758941    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:50:52.967692 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:52.966221    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"]
Oct 31 13:50:52.967692 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:52.966437    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26" podUID="17767cc6-6a5e-4bb9-84d2-98162742cd43" containerName="istio-proxy" containerID="cri-o://f2283b1a9fe3d8a84916184367cf889129bd985ebc494cc89c2ff9f3b6edeb20" gracePeriod=30
Oct 31 13:50:53.946444 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:53.946341    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"]
Oct 31 13:50:53.947706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:53.947359    4660 cpu_manager.go:414] "containerMap: removing container" podUID="5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" containerName="istio-proxy"
Oct 31 13:50:53.947706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:53.947382    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" containerName="istio-proxy"
Oct 31 13:50:53.947706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:53.947669    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" containerName="istio-proxy"
Oct 31 13:50:53.947706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:53.947694    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" containerName="istio-proxy"
Oct 31 13:50:53.959934 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:53.959912    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:53.962413 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:53.962384    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"]
Oct 31 13:50:53.966753 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:53.966728    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-33c10a\"/\"openshift-service-ca.crt\""
Oct 31 13:50:53.967809 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:53.967788    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-33c10a\"/\"default-dockercfg-trgbj\""
Oct 31 13:50:53.968325 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:53.968283    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-33c10a\"/\"kube-root-ca.crt\""
Oct 31 13:50:53.968508 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:53.968491    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-33c10a\"/\"rate-test-model-33c10a-kserve-self-signed-certs\""
Oct 31 13:50:54.085310 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.085255    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-dshm\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.085499 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.085393    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-tls-certs\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.085630 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.085610    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-tmp-dir\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.085724 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.085711    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-model-cache\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.085803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.085747    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n97kt\" (UniqueName: \"kubernetes.io/projected/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-kube-api-access-n97kt\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.085803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.085795    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-home\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.187607 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.187568    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-tmp-dir\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.189322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.189280    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-model-cache\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.200983 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.188665    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-tmp-dir\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.202625 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.189773    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-model-cache\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.202795 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.201435    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n97kt\" (UniqueName: \"kubernetes.io/projected/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-kube-api-access-n97kt\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.202932 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.202914    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-home\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.203188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.203172    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-dshm\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.203376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.203357    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-tls-certs\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.204586 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.204562    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-home\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.231366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.225600    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n97kt\" (UniqueName: \"kubernetes.io/projected/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-kube-api-access-n97kt\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.231366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.226919    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-tls-certs\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.232048 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.231994    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-dshm\") pod \"rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") " pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.291077 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.291044    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:54.633064 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:54.633036    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"]
Oct 31 13:50:54.635084 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:50:54.635041    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdb2b8e0_12fa_4b14_ac14_75e2aade9b80.slice/crio-edab8b1a162dd6fd853dc4b44a9f72b9a2cb39dd958c02ac4303539de9660943 WatchSource:0}: Error finding container edab8b1a162dd6fd853dc4b44a9f72b9a2cb39dd958c02ac4303539de9660943: Status 404 returned error can't find the container with id edab8b1a162dd6fd853dc4b44a9f72b9a2cb39dd958c02ac4303539de9660943
Oct 31 13:50:55.386708 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:55.386274    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b" event={"ID":"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80","Type":"ContainerStarted","Data":"5d11a1e1b8edcc4cb81b460d3d95ff8c56aaab5800f558bef177f2caa0b5d7b9"}
Oct 31 13:50:55.386708 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:55.386357    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b" event={"ID":"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80","Type":"ContainerStarted","Data":"edab8b1a162dd6fd853dc4b44a9f72b9a2cb39dd958c02ac4303539de9660943"}
Oct 31 13:50:55.415319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:55.414036    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b" podStartSLOduration=2.414017709 podStartE2EDuration="2.414017709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:50:55.410264103 +0000 UTC m=+2855.028923269" watchObservedRunningTime="2026-10-31 13:50:55.414017709 +0000 UTC m=+2855.032676874"
Oct 31 13:50:56.292021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:56.291985    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:50:58.415078 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.415036    4660 generic.go:356] "Generic (PLEG): container finished" podID="17767cc6-6a5e-4bb9-84d2-98162742cd43" containerID="f2283b1a9fe3d8a84916184367cf889129bd985ebc494cc89c2ff9f3b6edeb20" exitCode=0
Oct 31 13:50:58.415489 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.415104    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26" event={"ID":"17767cc6-6a5e-4bb9-84d2-98162742cd43","Type":"ContainerDied","Data":"f2283b1a9fe3d8a84916184367cf889129bd985ebc494cc89c2ff9f3b6edeb20"}
Oct 31 13:50:58.415489 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.415177    4660 scope.go:122] "RemoveContainer" containerID="42fc1641631e9d3f537382a83757b92c04964049e1ebe23efd0cec61d78408b8"
Oct 31 13:50:58.739469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.739434    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:50:58.917783 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.917740    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-podinfo\") pod \"17767cc6-6a5e-4bb9-84d2-98162742cd43\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") "
Oct 31 13:50:58.917974 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.917843    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-token\" (UniqueName: \"kubernetes.io/projected/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-token\") pod \"17767cc6-6a5e-4bb9-84d2-98162742cd43\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") "
Oct 31 13:50:58.917974 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.917885    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-workload-socket\") pod \"17767cc6-6a5e-4bb9-84d2-98162742cd43\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") "
Oct 31 13:50:58.917974 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.917967    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-workload-certs\") pod \"17767cc6-6a5e-4bb9-84d2-98162742cd43\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") "
Oct 31 13:50:58.918143 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.918019    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17767cc6-6a5e-4bb9-84d2-98162742cd43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17767cc6-6a5e-4bb9-84d2-98162742cd43-istiod-ca-cert\") pod \"17767cc6-6a5e-4bb9-84d2-98162742cd43\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") "
Oct 31 13:50:58.918143 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.918054    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-envoy\") pod \"17767cc6-6a5e-4bb9-84d2-98162742cd43\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") "
Oct 31 13:50:58.918143 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.918089    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-data\") pod \"17767cc6-6a5e-4bb9-84d2-98162742cd43\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") "
Oct 31 13:50:58.918143 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.918123    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17767cc6-6a5e-4bb9-84d2-98162742cd43-kube-api-access-9bfj8\" (UniqueName: \"kubernetes.io/projected/17767cc6-6a5e-4bb9-84d2-98162742cd43-kube-api-access-9bfj8\") pod \"17767cc6-6a5e-4bb9-84d2-98162742cd43\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") "
Oct 31 13:50:58.918372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.918284    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-credential-socket\") pod \"17767cc6-6a5e-4bb9-84d2-98162742cd43\" (UID: \"17767cc6-6a5e-4bb9-84d2-98162742cd43\") "
Oct 31 13:50:58.918428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.918389    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-data" pod "17767cc6-6a5e-4bb9-84d2-98162742cd43" (UID: "17767cc6-6a5e-4bb9-84d2-98162742cd43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:58.918835 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.918807    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17767cc6-6a5e-4bb9-84d2-98162742cd43-istiod-ca-cert" pod "17767cc6-6a5e-4bb9-84d2-98162742cd43" (UID: "17767cc6-6a5e-4bb9-84d2-98162742cd43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:50:58.919096 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.919051    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-credential-socket" pod "17767cc6-6a5e-4bb9-84d2-98162742cd43" (UID: "17767cc6-6a5e-4bb9-84d2-98162742cd43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:58.919312 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.919275    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-workload-socket" pod "17767cc6-6a5e-4bb9-84d2-98162742cd43" (UID: "17767cc6-6a5e-4bb9-84d2-98162742cd43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:58.919403 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.919322    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:58.919403 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.919346    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17767cc6-6a5e-4bb9-84d2-98162742cd43-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:58.919403 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.919361    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:58.919592 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.919479    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-workload-certs" pod "17767cc6-6a5e-4bb9-84d2-98162742cd43" (UID: "17767cc6-6a5e-4bb9-84d2-98162742cd43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:58.920205 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.920184    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-podinfo" pod "17767cc6-6a5e-4bb9-84d2-98162742cd43" (UID: "17767cc6-6a5e-4bb9-84d2-98162742cd43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:50:58.920470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.920444    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-token" pod "17767cc6-6a5e-4bb9-84d2-98162742cd43" (UID: "17767cc6-6a5e-4bb9-84d2-98162742cd43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:50:58.921010 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.920991    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17767cc6-6a5e-4bb9-84d2-98162742cd43-kube-api-access-9bfj8" pod "17767cc6-6a5e-4bb9-84d2-98162742cd43" (UID: "17767cc6-6a5e-4bb9-84d2-98162742cd43"). InnerVolumeSpecName "kube-api-access-9bfj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:50:58.921533 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:58.921508    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-envoy" pod "17767cc6-6a5e-4bb9-84d2-98162742cd43" (UID: "17767cc6-6a5e-4bb9-84d2-98162742cd43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:59.021012 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:59.020978    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:59.021012 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:59.021011    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:59.021223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:59.021022    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:59.021223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:59.021031    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:59.021223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:59.021040    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17767cc6-6a5e-4bb9-84d2-98162742cd43-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:59.021223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:59.021048    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bfj8\" (UniqueName: \"kubernetes.io/projected/17767cc6-6a5e-4bb9-84d2-98162742cd43-kube-api-access-9bfj8\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:59.441251 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:59.441209    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26" event={"ID":"17767cc6-6a5e-4bb9-84d2-98162742cd43","Type":"ContainerDied","Data":"a8b5ef69faaf065f03294c5acd285cc52f40ebb1a0ba4c9ab43d7e16e9934d70"}
Oct 31 13:50:59.441764 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:59.441232    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"
Oct 31 13:50:59.441764 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:59.441329    4660 scope.go:122] "RemoveContainer" containerID="f2283b1a9fe3d8a84916184367cf889129bd985ebc494cc89c2ff9f3b6edeb20"
Oct 31 13:50:59.498805 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:59.498763    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"]
Oct 31 13:50:59.507584 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:50:59.507549    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-2zj26"]
Oct 31 13:51:00.531212 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:00.531176    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17767cc6-6a5e-4bb9-84d2-98162742cd43" path="/var/lib/kubelet/pods/17767cc6-6a5e-4bb9-84d2-98162742cd43/volumes"
Oct 31 13:51:04.292521 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:04.292492    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:51:04.307092 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:04.307066    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:51:04.494880 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:04.494850    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:51:13.563466 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:13.563432    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"]
Oct 31 13:51:20.743868 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:51:20.743781    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:51:28.602508 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:28.602453    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z" podUID="c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9" containerName="main" containerID="cri-o://c974ad7b6be45d79c033a749587077eace33a22563c1d2c5e1a1c46e04ba30c6" gracePeriod=45
Oct 31 13:51:28.807860 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:28.804234    4660 generic.go:356] "Generic (PLEG): container finished" podID="c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9" containerID="c974ad7b6be45d79c033a749587077eace33a22563c1d2c5e1a1c46e04ba30c6" exitCode=0
Oct 31 13:51:28.807860 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:28.804321    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z" event={"ID":"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9","Type":"ContainerDied","Data":"c974ad7b6be45d79c033a749587077eace33a22563c1d2c5e1a1c46e04ba30c6"}
Oct 31 13:51:28.909969 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:28.909938    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:51:29.013764 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.013715    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-tmp-dir\") pod \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") "
Oct 31 13:51:29.013959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.013771    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-model-cache\") pod \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") "
Oct 31 13:51:29.013959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.013832    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-dshm\") pod \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") "
Oct 31 13:51:29.014087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.013960    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-tls-certs\") pod \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") "
Oct 31 13:51:29.014087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.014007    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-home\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-home\") pod \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") "
Oct 31 13:51:29.014087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.014045    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-model-cache" pod "c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9" (UID: "c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:51:29.014087 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.014054    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-tmp-dir" pod "c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9" (UID: "c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:51:29.014329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.014093    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-kube-api-access-79lzm\" (UniqueName: \"kubernetes.io/projected/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-kube-api-access-79lzm\") pod \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\" (UID: \"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9\") "
Oct 31 13:51:29.014329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.014266    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-home" pod "c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9" (UID: "c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:51:29.014693 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.014672    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:51:29.014773 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.014696    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:51:29.014773 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.014709    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:51:29.016687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.016658    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-tls-certs" pod "c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9" (UID: "c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:51:29.016687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.016680    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-dshm" pod "c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9" (UID: "c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:51:29.016843 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.016738    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-kube-api-access-79lzm" pod "c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9" (UID: "c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9"). InnerVolumeSpecName "kube-api-access-79lzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:51:29.116383 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.116347    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:51:29.116383 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.116384    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79lzm\" (UniqueName: \"kubernetes.io/projected/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-kube-api-access-79lzm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:51:29.116588 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.116399    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:51:29.665344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.665271    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"]
Oct 31 13:51:29.666268 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.666243    4660 cpu_manager.go:414] "containerMap: removing container" podUID="17767cc6-6a5e-4bb9-84d2-98162742cd43" containerName="istio-proxy"
Oct 31 13:51:29.666417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.666271    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17767cc6-6a5e-4bb9-84d2-98162742cd43" containerName="istio-proxy"
Oct 31 13:51:29.666417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.666323    4660 cpu_manager.go:414] "containerMap: removing container" podUID="17767cc6-6a5e-4bb9-84d2-98162742cd43" containerName="istio-proxy"
Oct 31 13:51:29.666417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.666344    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17767cc6-6a5e-4bb9-84d2-98162742cd43" containerName="istio-proxy"
Oct 31 13:51:29.666417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.666402    4660 cpu_manager.go:414] "containerMap: removing container" podUID="5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" containerName="istio-proxy"
Oct 31 13:51:29.666417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.666410    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bd757f7-0d36-4d30-a6ab-94e8f1b2cc9e" containerName="istio-proxy"
Oct 31 13:51:29.667215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.666435    4660 cpu_manager.go:414] "containerMap: removing container" podUID="c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9" containerName="main"
Oct 31 13:51:29.667215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.666443    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9" containerName="main"
Oct 31 13:51:29.667215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.666607    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="17767cc6-6a5e-4bb9-84d2-98162742cd43" containerName="istio-proxy"
Oct 31 13:51:29.667215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.666631    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9" containerName="main"
Oct 31 13:51:29.667215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.666646    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="17767cc6-6a5e-4bb9-84d2-98162742cd43" containerName="istio-proxy"
Oct 31 13:51:29.670975 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.670957    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.684853 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.683736    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"]
Oct 31 13:51:29.684853 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.684035    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-abdcb1\"/\"openshift-service-ca.crt\""
Oct 31 13:51:29.684853 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.684122    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-abdcb1\"/\"kube-root-ca.crt\""
Oct 31 13:51:29.685483 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.685463    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-abdcb1\"/\"default-dockercfg-r2bzq\""
Oct 31 13:51:29.685573 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.685513    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-abdcb1\"/\"rate-test-model-abdcb1-kserve-self-signed-certs\""
Oct 31 13:51:29.816846 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.816801    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z" event={"ID":"c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9","Type":"ContainerDied","Data":"8666a0c111be0f53c9c64646acf871225556f235e6876d35c1f6899f869f97ff"}
Oct 31 13:51:29.817027 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.816863    4660 scope.go:122] "RemoveContainer" containerID="c974ad7b6be45d79c033a749587077eace33a22563c1d2c5e1a1c46e04ba30c6"
Oct 31 13:51:29.817027 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.816817    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"
Oct 31 13:51:29.847414 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.833518    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j25q8\" (UniqueName: \"kubernetes.io/projected/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-kube-api-access-j25q8\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.847695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.847512    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-model-cache\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.847695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.847642    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-home\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.847695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.847677    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-tls-certs\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.847872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.847762    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-tmp-dir\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.847872 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.847797    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-dshm\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.881316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.877248    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"]
Oct 31 13:51:29.946375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.943936    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1ddade/test-model-1ddade-kserve-5c5bcc7bf-5487z"]
Oct 31 13:51:29.950313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.950023    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-dshm\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.950313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.950250    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j25q8\" (UniqueName: \"kubernetes.io/projected/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-kube-api-access-j25q8\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.950543 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.950366    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-model-cache\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.950543 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.950464    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-home\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.950543 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.950492    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-tls-certs\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.950700 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.950562    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-tmp-dir\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.963347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.951049    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-tmp-dir\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.963347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.953620    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-model-cache\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.963347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.954208    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-home\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.963347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.954560    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-dshm\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.963347 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.961583    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-tls-certs\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.978363 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.969029    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j25q8\" (UniqueName: \"kubernetes.io/projected/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-kube-api-access-j25q8\") pod \"rate-test-model-abdcb1-kserve-65d65d5f97-56h62\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") " pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:29.991669 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:29.990946    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:30.318369 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:30.317569    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"]
Oct 31 13:51:30.327241 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:51:30.327208    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b84ce8f_b99a_42bd_b684_ae26a52e8b3f.slice/crio-a2126d7c10934a5b968ed72113b9aa6f97b44ca5dd97f5543f40aca0adb64771 WatchSource:0}: Error finding container a2126d7c10934a5b968ed72113b9aa6f97b44ca5dd97f5543f40aca0adb64771: Status 404 returned error can't find the container with id a2126d7c10934a5b968ed72113b9aa6f97b44ca5dd97f5543f40aca0adb64771
Oct 31 13:51:30.531653 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:30.531613    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9" path="/var/lib/kubelet/pods/c7bcc2dc-c4b0-4d52-86bc-f5888f00a6d9/volumes"
Oct 31 13:51:30.826798 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:30.826761    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62" event={"ID":"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f","Type":"ContainerStarted","Data":"c2b1c45708beb0798c8cab1e182c12ff7d4e825553389f6a9fe3909b7c9c5d52"}
Oct 31 13:51:30.826798 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:30.826812    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62" event={"ID":"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f","Type":"ContainerStarted","Data":"a2126d7c10934a5b968ed72113b9aa6f97b44ca5dd97f5543f40aca0adb64771"}
Oct 31 13:51:30.851339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:30.851280    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62" podStartSLOduration=1.851267354 podStartE2EDuration="1.851267354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:51:30.84755327 +0000 UTC m=+2890.466212429" watchObservedRunningTime="2026-10-31 13:51:30.851267354 +0000 UTC m=+2890.469926560"
Oct 31 13:51:30.991995 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:30.991960    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:39.991375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:39.991326    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:40.006322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:40.006272    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:41.029469 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:41.029419    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:51:50.754672 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:51:50.754625    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:51:59.443877 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:59.443838    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"]
Oct 31 13:51:59.444827 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:51:59.444792    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv" podUID="3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c" containerName="maas-api" containerID="cri-o://724444f5e3bcf6a7025378e5075d391c707c1dcdb5d75a1ab4a67257960932ac" gracePeriod=30
Oct 31 13:52:00.068627 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.068599    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:52:00.195841 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.195800    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-openshift-service-ca\") pod \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\" (UID: \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\") "
Oct 31 13:52:00.196033 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.195969    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-maas-api-tls\") pod \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\" (UID: \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\") "
Oct 31 13:52:00.196109 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.196088    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-metrics-certs\") pod \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\" (UID: \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\") "
Oct 31 13:52:00.196274 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.196253    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-kube-api-access-chbzp\" (UniqueName: \"kubernetes.io/projected/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-kube-api-access-chbzp\") pod \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\" (UID: \"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c\") "
Oct 31 13:52:00.196406 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.196275    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-openshift-service-ca" pod "3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c" (UID: "3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:52:00.196864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.196840    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:00.202310 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.201719    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-metrics-certs" pod "3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c" (UID: "3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:52:00.214739 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.213461    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-kube-api-access-chbzp" pod "3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c" (UID: "3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c"). InnerVolumeSpecName "kube-api-access-chbzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:52:00.237032 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.235466    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-maas-api-tls" pod "3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c" (UID: "3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:52:00.261614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.261203    4660 generic.go:356] "Generic (PLEG): container finished" podID="3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c" containerID="724444f5e3bcf6a7025378e5075d391c707c1dcdb5d75a1ab4a67257960932ac" exitCode=0
Oct 31 13:52:00.261614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.261475    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv" event={"ID":"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c","Type":"ContainerDied","Data":"724444f5e3bcf6a7025378e5075d391c707c1dcdb5d75a1ab4a67257960932ac"}
Oct 31 13:52:00.261614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.261530    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv" event={"ID":"3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c","Type":"ContainerDied","Data":"7e5a5620b3c05db741f6f4642b16d707604ef63df90dcffeb5fc7e7f7a9d665c"}
Oct 31 13:52:00.261614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.261561    4660 scope.go:122] "RemoveContainer" containerID="724444f5e3bcf6a7025378e5075d391c707c1dcdb5d75a1ab4a67257960932ac"
Oct 31 13:52:00.262176 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.261795    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"
Oct 31 13:52:00.280094 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.280069    4660 scope.go:122] "RemoveContainer" containerID="724444f5e3bcf6a7025378e5075d391c707c1dcdb5d75a1ab4a67257960932ac"
Oct 31 13:52:00.280451 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:52:00.280425    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724444f5e3bcf6a7025378e5075d391c707c1dcdb5d75a1ab4a67257960932ac\": container with ID starting with 724444f5e3bcf6a7025378e5075d391c707c1dcdb5d75a1ab4a67257960932ac not found: ID does not exist" containerID="724444f5e3bcf6a7025378e5075d391c707c1dcdb5d75a1ab4a67257960932ac"
Oct 31 13:52:00.280555 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.280462    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724444f5e3bcf6a7025378e5075d391c707c1dcdb5d75a1ab4a67257960932ac"} err="failed to get container status \"724444f5e3bcf6a7025378e5075d391c707c1dcdb5d75a1ab4a67257960932ac\": rpc error: code = NotFound desc = could not find container \"724444f5e3bcf6a7025378e5075d391c707c1dcdb5d75a1ab4a67257960932ac\": container with ID starting with 724444f5e3bcf6a7025378e5075d391c707c1dcdb5d75a1ab4a67257960932ac not found: ID does not exist"
Oct 31 13:52:00.294858 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.294829    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"]
Oct 31 13:52:00.300130 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.299566    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:00.300130 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.299606    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:00.300130 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.299634    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chbzp\" (UniqueName: \"kubernetes.io/projected/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c-kube-api-access-chbzp\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:00.309052 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.309020    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1ddade-66f6f66749-jj5xv"]
Oct 31 13:52:00.533670 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:00.533583    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c" path="/var/lib/kubelet/pods/3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c/volumes"
Oct 31 13:52:07.654468 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:07.654426    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"]
Oct 31 13:52:07.655105 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:07.654627    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn" podUID="1ea4837c-09ce-427d-9a97-49c7dd077463" containerName="istio-proxy" containerID="cri-o://0122eb68071742133ac0d6a2e59f997bf5bb1b099e9bc94c6f96b95d81d63c9f" gracePeriod=30
Oct 31 13:52:12.979092 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:12.979062    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:52:13.036464 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.036425    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-podinfo\") pod \"1ea4837c-09ce-427d-9a97-49c7dd077463\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") "
Oct 31 13:52:13.036652 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.036608    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-token\" (UniqueName: \"kubernetes.io/projected/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-token\") pod \"1ea4837c-09ce-427d-9a97-49c7dd077463\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") "
Oct 31 13:52:13.036722 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.036682    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-credential-socket\") pod \"1ea4837c-09ce-427d-9a97-49c7dd077463\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") "
Oct 31 13:52:13.036785 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.036720    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea4837c-09ce-427d-9a97-49c7dd077463-kube-api-access-pfvmm\" (UniqueName: \"kubernetes.io/projected/1ea4837c-09ce-427d-9a97-49c7dd077463-kube-api-access-pfvmm\") pod \"1ea4837c-09ce-427d-9a97-49c7dd077463\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") "
Oct 31 13:52:13.036785 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.036750    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-envoy\") pod \"1ea4837c-09ce-427d-9a97-49c7dd077463\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") "
Oct 31 13:52:13.036909 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.036855    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-workload-certs\") pod \"1ea4837c-09ce-427d-9a97-49c7dd077463\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") "
Oct 31 13:52:13.036973 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.036912    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-workload-socket\") pod \"1ea4837c-09ce-427d-9a97-49c7dd077463\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") "
Oct 31 13:52:13.036973 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.036962    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea4837c-09ce-427d-9a97-49c7dd077463-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea4837c-09ce-427d-9a97-49c7dd077463-istiod-ca-cert\") pod \"1ea4837c-09ce-427d-9a97-49c7dd077463\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") "
Oct 31 13:52:13.037117 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.037025    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-data\") pod \"1ea4837c-09ce-427d-9a97-49c7dd077463\" (UID: \"1ea4837c-09ce-427d-9a97-49c7dd077463\") "
Oct 31 13:52:13.038327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.038283    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-workload-certs" pod "1ea4837c-09ce-427d-9a97-49c7dd077463" (UID: "1ea4837c-09ce-427d-9a97-49c7dd077463"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:52:13.038550 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.038416    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea4837c-09ce-427d-9a97-49c7dd077463-istiod-ca-cert" pod "1ea4837c-09ce-427d-9a97-49c7dd077463" (UID: "1ea4837c-09ce-427d-9a97-49c7dd077463"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:52:13.038789 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.038761    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-workload-socket" pod "1ea4837c-09ce-427d-9a97-49c7dd077463" (UID: "1ea4837c-09ce-427d-9a97-49c7dd077463"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:52:13.038937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.038911    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-credential-socket" pod "1ea4837c-09ce-427d-9a97-49c7dd077463" (UID: "1ea4837c-09ce-427d-9a97-49c7dd077463"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:52:13.039009 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.038993    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-data" pod "1ea4837c-09ce-427d-9a97-49c7dd077463" (UID: "1ea4837c-09ce-427d-9a97-49c7dd077463"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:52:13.041263 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.041220    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-envoy" pod "1ea4837c-09ce-427d-9a97-49c7dd077463" (UID: "1ea4837c-09ce-427d-9a97-49c7dd077463"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:52:13.041576 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.041454    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-token" pod "1ea4837c-09ce-427d-9a97-49c7dd077463" (UID: "1ea4837c-09ce-427d-9a97-49c7dd077463"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:52:13.041697 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.041672    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-podinfo" pod "1ea4837c-09ce-427d-9a97-49c7dd077463" (UID: "1ea4837c-09ce-427d-9a97-49c7dd077463"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:52:13.044457 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.044429    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea4837c-09ce-427d-9a97-49c7dd077463-kube-api-access-pfvmm" pod "1ea4837c-09ce-427d-9a97-49c7dd077463" (UID: "1ea4837c-09ce-427d-9a97-49c7dd077463"). InnerVolumeSpecName "kube-api-access-pfvmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:52:13.139256 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.139222    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.139256 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.139252    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfvmm\" (UniqueName: \"kubernetes.io/projected/1ea4837c-09ce-427d-9a97-49c7dd077463-kube-api-access-pfvmm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.139256 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.139262    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.139601 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.139271    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.139601 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.139282    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.139601 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.139305    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea4837c-09ce-427d-9a97-49c7dd077463-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.139601 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.139318    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.139601 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.139327    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.139601 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.139335    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ea4837c-09ce-427d-9a97-49c7dd077463-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.382388 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.382353    4660 generic.go:356] "Generic (PLEG): container finished" podID="1ea4837c-09ce-427d-9a97-49c7dd077463" containerID="0122eb68071742133ac0d6a2e59f997bf5bb1b099e9bc94c6f96b95d81d63c9f" exitCode=0
Oct 31 13:52:13.382584 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.382413    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"
Oct 31 13:52:13.382584 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.382433    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn" event={"ID":"1ea4837c-09ce-427d-9a97-49c7dd077463","Type":"ContainerDied","Data":"0122eb68071742133ac0d6a2e59f997bf5bb1b099e9bc94c6f96b95d81d63c9f"}
Oct 31 13:52:13.382584 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.382492    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn" event={"ID":"1ea4837c-09ce-427d-9a97-49c7dd077463","Type":"ContainerDied","Data":"68e7f1447fc627d66546ccb9df11fd6b317bfaad2edc802f9125d92e3e54dab4"}
Oct 31 13:52:13.382584 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.382518    4660 scope.go:122] "RemoveContainer" containerID="0122eb68071742133ac0d6a2e59f997bf5bb1b099e9bc94c6f96b95d81d63c9f"
Oct 31 13:52:13.394260 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.394241    4660 scope.go:122] "RemoveContainer" containerID="0122eb68071742133ac0d6a2e59f997bf5bb1b099e9bc94c6f96b95d81d63c9f"
Oct 31 13:52:13.394568 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:52:13.394548    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0122eb68071742133ac0d6a2e59f997bf5bb1b099e9bc94c6f96b95d81d63c9f\": container with ID starting with 0122eb68071742133ac0d6a2e59f997bf5bb1b099e9bc94c6f96b95d81d63c9f not found: ID does not exist" containerID="0122eb68071742133ac0d6a2e59f997bf5bb1b099e9bc94c6f96b95d81d63c9f"
Oct 31 13:52:13.394655 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.394576    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0122eb68071742133ac0d6a2e59f997bf5bb1b099e9bc94c6f96b95d81d63c9f"} err="failed to get container status \"0122eb68071742133ac0d6a2e59f997bf5bb1b099e9bc94c6f96b95d81d63c9f\": rpc error: code = NotFound desc = could not find container \"0122eb68071742133ac0d6a2e59f997bf5bb1b099e9bc94c6f96b95d81d63c9f\": container with ID starting with 0122eb68071742133ac0d6a2e59f997bf5bb1b099e9bc94c6f96b95d81d63c9f not found: ID does not exist"
Oct 31 13:52:13.409821 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.409778    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"]
Oct 31 13:52:13.414136 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:13.414115    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-nx9hn"]
Oct 31 13:52:14.533585 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:14.533538    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea4837c-09ce-427d-9a97-49c7dd077463" path="/var/lib/kubelet/pods/1ea4837c-09ce-427d-9a97-49c7dd077463/volumes"
Oct 31 13:52:20.737332 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:52:20.736277    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:52:21.076707 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.076623    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"]
Oct 31 13:52:21.077270 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.077253    4660 cpu_manager.go:414] "containerMap: removing container" podUID="1ea4837c-09ce-427d-9a97-49c7dd077463" containerName="istio-proxy"
Oct 31 13:52:21.077356 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.077273    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea4837c-09ce-427d-9a97-49c7dd077463" containerName="istio-proxy"
Oct 31 13:52:21.077356 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.077308    4660 cpu_manager.go:414] "containerMap: removing container" podUID="3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c" containerName="maas-api"
Oct 31 13:52:21.077356 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.077315    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c" containerName="maas-api"
Oct 31 13:52:21.077471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.077445    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a5f4f4d-eb62-44aa-a8e2-ab0efd03617c" containerName="maas-api"
Oct 31 13:52:21.077471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.077467    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea4837c-09ce-427d-9a97-49c7dd077463" containerName="istio-proxy"
Oct 31 13:52:21.081085 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.081068    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.088498 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.088471    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-abdcb1-openshift-default-dockercfg-jnc5d\""
Oct 31 13:52:21.094580 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.094555    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"]
Oct 31 13:52:21.175749 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.175712    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-podinfo\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.175911 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.175797    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istiod-ca-cert\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.175911 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.175822    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q87f4\" (UniqueName: \"kubernetes.io/projected/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-kube-api-access-q87f4\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.175911 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.175873    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-workload-certs\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.176027 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.175983    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-credential-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.176027 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.176006    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-envoy\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.176184 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.176156    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-data\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.176223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.176204    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-workload-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.176306 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.176283    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-token\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.277587 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.277551    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-workload-certs\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.277806 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.277608    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-credential-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.277806 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.277634    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-envoy\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.277806 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.277731    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-data\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.277806 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.277767    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-workload-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.278048 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.277834    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-token\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.278048 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.277981    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-podinfo\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.278048 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.277998    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-workload-certs\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.278220 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.278047    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-credential-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.278220 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.278056    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istiod-ca-cert\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.278220 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.278124    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87f4\" (UniqueName: \"kubernetes.io/projected/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-kube-api-access-q87f4\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.278675 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.278655    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-data\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.278767 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.278670    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istiod-ca-cert\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.278767 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.278715    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-workload-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.280541 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.280519    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-podinfo\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.280842 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.280822    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-envoy\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.294091 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.294064    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-token\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.294335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.294289    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q87f4\" (UniqueName: \"kubernetes.io/projected/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-kube-api-access-q87f4\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.398166 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.398131    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:21.692944 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.692917    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"]
Oct 31 13:52:21.693667 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:52:21.693639    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5549d7d1_08ba_419c_8cf1_9c2b29ceaec3.slice/crio-845ccf7eb89c60a0bf3ebf4af703be265b7a1419a4965ce4a3fe6080acff4c56 WatchSource:0}: Error finding container 845ccf7eb89c60a0bf3ebf4af703be265b7a1419a4965ce4a3fe6080acff4c56: Status 404 returned error can't find the container with id 845ccf7eb89c60a0bf3ebf4af703be265b7a1419a4965ce4a3fe6080acff4c56
Oct 31 13:52:21.696979 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.696945    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:52:21.697055 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.697024    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:52:21.697094 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:21.697053    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:52:22.474178 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:22.474143    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j" event={"ID":"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3","Type":"ContainerStarted","Data":"2ce6dad9f56edee1449510a709103a7a5d078b7a9c18279f6218974a9804c3da"}
Oct 31 13:52:22.474689 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:22.474202    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j" event={"ID":"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3","Type":"ContainerStarted","Data":"845ccf7eb89c60a0bf3ebf4af703be265b7a1419a4965ce4a3fe6080acff4c56"}
Oct 31 13:52:22.503272 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:22.503212    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j" podStartSLOduration=1.5031939460000001 podStartE2EDuration="1.503193946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:52:22.498365951 +0000 UTC m=+2942.117025126" watchObservedRunningTime="2026-10-31 13:52:22.503193946 +0000 UTC m=+2942.121853111"
Oct 31 13:52:23.399213 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:23.399180    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:23.402928 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:23.400864    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 31 13:52:23.402928 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:23.400931    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j" podUID="5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 31 13:52:24.399223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:24.399185    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 31 13:52:24.399685 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:24.399247    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j" podUID="5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 31 13:52:25.447331 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:25.443977    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 31 13:52:25.447331 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:25.444071    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j" podUID="5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 31 13:52:26.399456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:26.399418    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 31 13:52:26.399691 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:26.399492    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j" podUID="5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 31 13:52:27.398706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:27.398664    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 31 13:52:27.399211 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:27.398732    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j" podUID="5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 31 13:52:28.532749 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:28.532702    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:28.533375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:28.533350    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:28.534370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:52:28.534348    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 13:52:50.748661 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:52:50.748612    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:53:13.342171 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:13.342136    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"]
Oct 31 13:53:13.342719 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:13.342364    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt" podUID="ede7a814-d7a4-44ef-b0b4-bf7377741d13" containerName="istio-proxy" containerID="cri-o://ad35002e2b3dac940ba41aff72582a11e6b5da903574f5efd1fee4f78c581b96" gracePeriod=30
Oct 31 13:53:16.964497 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:16.964462    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"]
Oct 31 13:53:16.964920 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:16.964664    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8" podUID="9d3e7468-c108-44b5-8fae-a38ddc3d2e42" containerName="istio-proxy" containerID="cri-o://7c687929ea3cc33327ed487de305acb450c50d8bbd0adbd6986fe62d59abe612" gracePeriod=30
Oct 31 13:53:18.636214 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.636183    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:53:18.753458 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.753426    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-envoy\") pod \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") "
Oct 31 13:53:18.753615 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.753552    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istiod-ca-cert\") pod \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") "
Oct 31 13:53:18.753679 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.753656    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-workload-socket\") pod \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") "
Oct 31 13:53:18.753732 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.753691    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-token\" (UniqueName: \"kubernetes.io/projected/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-token\") pod \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") "
Oct 31 13:53:18.753796 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.753728    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-data\") pod \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") "
Oct 31 13:53:18.753877 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.753847    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-credential-socket\") pod \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") "
Oct 31 13:53:18.753990 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.753930    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede7a814-d7a4-44ef-b0b4-bf7377741d13-kube-api-access-6bnzj\" (UniqueName: \"kubernetes.io/projected/ede7a814-d7a4-44ef-b0b4-bf7377741d13-kube-api-access-6bnzj\") pod \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") "
Oct 31 13:53:18.753990 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.753976    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-workload-certs\") pod \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") "
Oct 31 13:53:18.754150 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.754134    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-podinfo\") pod \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\" (UID: \"ede7a814-d7a4-44ef-b0b4-bf7377741d13\") "
Oct 31 13:53:18.754960 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.754698    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-credential-socket" pod "ede7a814-d7a4-44ef-b0b4-bf7377741d13" (UID: "ede7a814-d7a4-44ef-b0b4-bf7377741d13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:18.754960 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.754766    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-workload-socket" pod "ede7a814-d7a4-44ef-b0b4-bf7377741d13" (UID: "ede7a814-d7a4-44ef-b0b4-bf7377741d13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:18.754960 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.754925    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-workload-certs" pod "ede7a814-d7a4-44ef-b0b4-bf7377741d13" (UID: "ede7a814-d7a4-44ef-b0b4-bf7377741d13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:18.755189 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.755078    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:18.755189 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.755098    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:18.755189 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.755112    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:18.755402 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.755266    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istiod-ca-cert" pod "ede7a814-d7a4-44ef-b0b4-bf7377741d13" (UID: "ede7a814-d7a4-44ef-b0b4-bf7377741d13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:53:18.755634 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.755610    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-data" pod "ede7a814-d7a4-44ef-b0b4-bf7377741d13" (UID: "ede7a814-d7a4-44ef-b0b4-bf7377741d13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:18.756089 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.756068    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-envoy" pod "ede7a814-d7a4-44ef-b0b4-bf7377741d13" (UID: "ede7a814-d7a4-44ef-b0b4-bf7377741d13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:18.756783 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.756757    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-token" pod "ede7a814-d7a4-44ef-b0b4-bf7377741d13" (UID: "ede7a814-d7a4-44ef-b0b4-bf7377741d13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:53:18.757192 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.757174    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-podinfo" pod "ede7a814-d7a4-44ef-b0b4-bf7377741d13" (UID: "ede7a814-d7a4-44ef-b0b4-bf7377741d13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:53:18.757837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.757806    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede7a814-d7a4-44ef-b0b4-bf7377741d13-kube-api-access-6bnzj" pod "ede7a814-d7a4-44ef-b0b4-bf7377741d13" (UID: "ede7a814-d7a4-44ef-b0b4-bf7377741d13"). InnerVolumeSpecName "kube-api-access-6bnzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:53:18.857054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.857025    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:18.857054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.857053    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:18.857054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.857062    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:18.857332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.857071    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:18.857332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.857080    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede7a814-d7a4-44ef-b0b4-bf7377741d13-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:18.857332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:18.857090    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bnzj\" (UniqueName: \"kubernetes.io/projected/ede7a814-d7a4-44ef-b0b4-bf7377741d13-kube-api-access-6bnzj\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:19.237371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:19.237321    4660 generic.go:356] "Generic (PLEG): container finished" podID="ede7a814-d7a4-44ef-b0b4-bf7377741d13" containerID="ad35002e2b3dac940ba41aff72582a11e6b5da903574f5efd1fee4f78c581b96" exitCode=0
Oct 31 13:53:19.237571 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:19.237369    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt" event={"ID":"ede7a814-d7a4-44ef-b0b4-bf7377741d13","Type":"ContainerDied","Data":"ad35002e2b3dac940ba41aff72582a11e6b5da903574f5efd1fee4f78c581b96"}
Oct 31 13:53:19.237571 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:19.237444    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt" event={"ID":"ede7a814-d7a4-44ef-b0b4-bf7377741d13","Type":"ContainerDied","Data":"962e0dcae352d166000290a22260239b0a10cb4dcf29955fab37ddf6405ff0e8"}
Oct 31 13:53:19.237571 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:19.237390    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"
Oct 31 13:53:19.237571 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:19.237487    4660 scope.go:122] "RemoveContainer" containerID="ad35002e2b3dac940ba41aff72582a11e6b5da903574f5efd1fee4f78c581b96"
Oct 31 13:53:19.252641 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:19.252623    4660 scope.go:122] "RemoveContainer" containerID="ad35002e2b3dac940ba41aff72582a11e6b5da903574f5efd1fee4f78c581b96"
Oct 31 13:53:19.252942 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:53:19.252920    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad35002e2b3dac940ba41aff72582a11e6b5da903574f5efd1fee4f78c581b96\": container with ID starting with ad35002e2b3dac940ba41aff72582a11e6b5da903574f5efd1fee4f78c581b96 not found: ID does not exist" containerID="ad35002e2b3dac940ba41aff72582a11e6b5da903574f5efd1fee4f78c581b96"
Oct 31 13:53:19.253033 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:19.252948    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad35002e2b3dac940ba41aff72582a11e6b5da903574f5efd1fee4f78c581b96"} err="failed to get container status \"ad35002e2b3dac940ba41aff72582a11e6b5da903574f5efd1fee4f78c581b96\": rpc error: code = NotFound desc = could not find container \"ad35002e2b3dac940ba41aff72582a11e6b5da903574f5efd1fee4f78c581b96\": container with ID starting with ad35002e2b3dac940ba41aff72582a11e6b5da903574f5efd1fee4f78c581b96 not found: ID does not exist"
Oct 31 13:53:19.268466 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:19.268443    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"]
Oct 31 13:53:19.279486 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:19.279465    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g4gmt"]
Oct 31 13:53:20.532932 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:20.532891    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede7a814-d7a4-44ef-b0b4-bf7377741d13" path="/var/lib/kubelet/pods/ede7a814-d7a4-44ef-b0b4-bf7377741d13/volumes"
Oct 31 13:53:20.734977 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:53:20.734936    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:53:20.804973 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:20.804889    4660 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:53:20.804973 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:20.804947    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8" podUID="9d3e7468-c108-44b5-8fae-a38ddc3d2e42" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:53:21.233702 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:21.233669    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:53:21.249578 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:21.249545    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:53:21.253148 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:21.253122    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:53:21.259537 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:21.259511    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:53:21.265831 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:21.265808    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:53:21.275537 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:21.275515    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:53:22.269523 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.269377    4660 generic.go:356] "Generic (PLEG): container finished" podID="9d3e7468-c108-44b5-8fae-a38ddc3d2e42" containerID="7c687929ea3cc33327ed487de305acb450c50d8bbd0adbd6986fe62d59abe612" exitCode=0
Oct 31 13:53:22.269523 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.269458    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8" event={"ID":"9d3e7468-c108-44b5-8fae-a38ddc3d2e42","Type":"ContainerDied","Data":"7c687929ea3cc33327ed487de305acb450c50d8bbd0adbd6986fe62d59abe612"}
Oct 31 13:53:22.316058 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.315708    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:53:22.359236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.359203    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-kube-api-access-b7clb\" (UniqueName: \"kubernetes.io/projected/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-kube-api-access-b7clb\") pod \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") "
Oct 31 13:53:22.359418 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.359252    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-token\" (UniqueName: \"kubernetes.io/projected/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-token\") pod \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") "
Oct 31 13:53:22.359467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.359438    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-envoy\") pod \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") "
Oct 31 13:53:22.359517 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.359482    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-data\") pod \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") "
Oct 31 13:53:22.359687 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.359666    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-workload-certs\") pod \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") "
Oct 31 13:53:22.359878 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.359722    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-workload-socket\") pod \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") "
Oct 31 13:53:22.359969 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.359915    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-workload-certs" pod "9d3e7468-c108-44b5-8fae-a38ddc3d2e42" (UID: "9d3e7468-c108-44b5-8fae-a38ddc3d2e42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.360038 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.360001    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-workload-socket" pod "9d3e7468-c108-44b5-8fae-a38ddc3d2e42" (UID: "9d3e7468-c108-44b5-8fae-a38ddc3d2e42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.360110 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.360065    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-credential-socket\") pod \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") "
Oct 31 13:53:22.360171 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.360142    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-podinfo\") pod \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") "
Oct 31 13:53:22.360249 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.360225    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-credential-socket" pod "9d3e7468-c108-44b5-8fae-a38ddc3d2e42" (UID: "9d3e7468-c108-44b5-8fae-a38ddc3d2e42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.360333 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.360234    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istiod-ca-cert\") pod \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\" (UID: \"9d3e7468-c108-44b5-8fae-a38ddc3d2e42\") "
Oct 31 13:53:22.360551 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.360530    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istiod-ca-cert" pod "9d3e7468-c108-44b5-8fae-a38ddc3d2e42" (UID: "9d3e7468-c108-44b5-8fae-a38ddc3d2e42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:53:22.361271 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.361244    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.361396 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.361274    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.361396 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.361305    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.361396 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.361320    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.362026 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.361990    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-kube-api-access-b7clb" pod "9d3e7468-c108-44b5-8fae-a38ddc3d2e42" (UID: "9d3e7468-c108-44b5-8fae-a38ddc3d2e42"). InnerVolumeSpecName "kube-api-access-b7clb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:53:22.362511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.362486    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-podinfo" pod "9d3e7468-c108-44b5-8fae-a38ddc3d2e42" (UID: "9d3e7468-c108-44b5-8fae-a38ddc3d2e42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:53:22.362511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.362500    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-data" pod "9d3e7468-c108-44b5-8fae-a38ddc3d2e42" (UID: "9d3e7468-c108-44b5-8fae-a38ddc3d2e42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.363484 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.363454    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-token" pod "9d3e7468-c108-44b5-8fae-a38ddc3d2e42" (UID: "9d3e7468-c108-44b5-8fae-a38ddc3d2e42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:53:22.363722 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.363702    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-envoy" pod "9d3e7468-c108-44b5-8fae-a38ddc3d2e42" (UID: "9d3e7468-c108-44b5-8fae-a38ddc3d2e42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.466913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.466878    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.466913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.466913    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7clb\" (UniqueName: \"kubernetes.io/projected/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-kube-api-access-b7clb\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.467132 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.466928    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.467132 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.466944    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.467132 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:22.466956    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3e7468-c108-44b5-8fae-a38ddc3d2e42-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:23.280913 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:23.280881    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"
Oct 31 13:53:23.281438 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:23.280878    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8" event={"ID":"9d3e7468-c108-44b5-8fae-a38ddc3d2e42","Type":"ContainerDied","Data":"8d1392fb76d3e55cd560d1b90a3b32f64e6b3d6834cae835a848c7b6d8bc94dc"}
Oct 31 13:53:23.281438 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:23.281102    4660 scope.go:122] "RemoveContainer" containerID="7c687929ea3cc33327ed487de305acb450c50d8bbd0adbd6986fe62d59abe612"
Oct 31 13:53:23.310035 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:23.310008    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"]
Oct 31 13:53:23.318440 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:23.318410    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-swsf8"]
Oct 31 13:53:24.532905 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:53:24.532872    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3e7468-c108-44b5-8fae-a38ddc3d2e42" path="/var/lib/kubelet/pods/9d3e7468-c108-44b5-8fae-a38ddc3d2e42/volumes"
Oct 31 13:53:50.765680 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:53:50.765589    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:54:20.743974 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:54:20.743936    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:54:50.753023 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:54:50.752981    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:54:53.110657 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.110613    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"]
Oct 31 13:54:53.111597 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.111553    4660 cpu_manager.go:414] "containerMap: removing container" podUID="9d3e7468-c108-44b5-8fae-a38ddc3d2e42" containerName="istio-proxy"
Oct 31 13:54:53.111790 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.111768    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3e7468-c108-44b5-8fae-a38ddc3d2e42" containerName="istio-proxy"
Oct 31 13:54:53.111949 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.111937    4660 cpu_manager.go:414] "containerMap: removing container" podUID="ede7a814-d7a4-44ef-b0b4-bf7377741d13" containerName="istio-proxy"
Oct 31 13:54:53.112041 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.112029    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede7a814-d7a4-44ef-b0b4-bf7377741d13" containerName="istio-proxy"
Oct 31 13:54:53.119586 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.119558    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3e7468-c108-44b5-8fae-a38ddc3d2e42" containerName="istio-proxy"
Oct 31 13:54:53.119765 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.119752    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede7a814-d7a4-44ef-b0b4-bf7377741d13" containerName="istio-proxy"
Oct 31 13:54:53.124865 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.124841    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"]
Oct 31 13:54:53.126420 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.126256    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"
Oct 31 13:54:53.136880 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.136859    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:54:53.137091 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.137067    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:54:53.207885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.207842    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp8l9\" (UniqueName: \"kubernetes.io/projected/bcd0752f-b6ee-4fa6-8084-574035cffcb3-kube-api-access-pp8l9\") pod \"maas-api-revoke-sub-c795281d-z4zjn\" (UID: \"bcd0752f-b6ee-4fa6-8084-574035cffcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"
Oct 31 13:54:53.215881 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.215848    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd0752f-b6ee-4fa6-8084-574035cffcb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c795281d-z4zjn\" (UID: \"bcd0752f-b6ee-4fa6-8084-574035cffcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"
Oct 31 13:54:53.216385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.216279    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd0752f-b6ee-4fa6-8084-574035cffcb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c795281d-z4zjn\" (UID: \"bcd0752f-b6ee-4fa6-8084-574035cffcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"
Oct 31 13:54:53.317701 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.317661    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp8l9\" (UniqueName: \"kubernetes.io/projected/bcd0752f-b6ee-4fa6-8084-574035cffcb3-kube-api-access-pp8l9\") pod \"maas-api-revoke-sub-c795281d-z4zjn\" (UID: \"bcd0752f-b6ee-4fa6-8084-574035cffcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"
Oct 31 13:54:53.317922 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.317804    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd0752f-b6ee-4fa6-8084-574035cffcb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c795281d-z4zjn\" (UID: \"bcd0752f-b6ee-4fa6-8084-574035cffcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"
Oct 31 13:54:53.318001 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.317954    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd0752f-b6ee-4fa6-8084-574035cffcb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c795281d-z4zjn\" (UID: \"bcd0752f-b6ee-4fa6-8084-574035cffcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"
Oct 31 13:54:53.318694 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.318662    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd0752f-b6ee-4fa6-8084-574035cffcb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c795281d-z4zjn\" (UID: \"bcd0752f-b6ee-4fa6-8084-574035cffcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"
Oct 31 13:54:53.322702 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.322654    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd0752f-b6ee-4fa6-8084-574035cffcb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c795281d-z4zjn\" (UID: \"bcd0752f-b6ee-4fa6-8084-574035cffcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"
Oct 31 13:54:53.333451 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.333422    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp8l9\" (UniqueName: \"kubernetes.io/projected/bcd0752f-b6ee-4fa6-8084-574035cffcb3-kube-api-access-pp8l9\") pod \"maas-api-revoke-sub-c795281d-z4zjn\" (UID: \"bcd0752f-b6ee-4fa6-8084-574035cffcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"
Oct 31 13:54:53.457866 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.457828    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"
Oct 31 13:54:53.771382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:53.771349    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"]
Oct 31 13:54:53.771727 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:54:53.771696    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd0752f_b6ee_4fa6_8084_574035cffcb3.slice/crio-0ae3beeca25fc53fa5db7478680aa7c4361f947ae7cdbbe4965659e51b531b20 WatchSource:0}: Error finding container 0ae3beeca25fc53fa5db7478680aa7c4361f947ae7cdbbe4965659e51b531b20: Status 404 returned error can't find the container with id 0ae3beeca25fc53fa5db7478680aa7c4361f947ae7cdbbe4965659e51b531b20
Oct 31 13:54:54.400798 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:54.400759    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn" event={"ID":"bcd0752f-b6ee-4fa6-8084-574035cffcb3","Type":"ContainerStarted","Data":"c14a75a5800897db4cb158c19a4934bd8f28d5116a168393f1a3cc6c4958ad40"}
Oct 31 13:54:54.401345 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:54.400816    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn" event={"ID":"bcd0752f-b6ee-4fa6-8084-574035cffcb3","Type":"ContainerStarted","Data":"0ae3beeca25fc53fa5db7478680aa7c4361f947ae7cdbbe4965659e51b531b20"}
Oct 31 13:54:54.442800 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:54.442729    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn" podStartSLOduration=1.442709901 podStartE2EDuration="1.442709901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:54.437690732 +0000 UTC m=+3094.056349895" watchObservedRunningTime="2026-10-31 13:54:54.442709901 +0000 UTC m=+3094.061369065"
Oct 31 13:54:55.434013 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:55.433971    4660 generic.go:356] "Generic (PLEG): container finished" podID="bcd0752f-b6ee-4fa6-8084-574035cffcb3" containerID="c14a75a5800897db4cb158c19a4934bd8f28d5116a168393f1a3cc6c4958ad40" exitCode=0
Oct 31 13:54:55.434746 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:55.434708    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn" event={"ID":"bcd0752f-b6ee-4fa6-8084-574035cffcb3","Type":"ContainerDied","Data":"c14a75a5800897db4cb158c19a4934bd8f28d5116a168393f1a3cc6c4958ad40"}
Oct 31 13:54:56.647743 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:56.647706    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"
Oct 31 13:54:56.694340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:56.694233    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd0752f-b6ee-4fa6-8084-574035cffcb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd0752f-b6ee-4fa6-8084-574035cffcb3-maas-api-openshift-ca\") pod \"bcd0752f-b6ee-4fa6-8084-574035cffcb3\" (UID: \"bcd0752f-b6ee-4fa6-8084-574035cffcb3\") "
Oct 31 13:54:56.694507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:56.694417    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcd0752f-b6ee-4fa6-8084-574035cffcb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd0752f-b6ee-4fa6-8084-574035cffcb3-maas-api-xks-ca\") pod \"bcd0752f-b6ee-4fa6-8084-574035cffcb3\" (UID: \"bcd0752f-b6ee-4fa6-8084-574035cffcb3\") "
Oct 31 13:54:56.694586 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:56.694527    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd0752f-b6ee-4fa6-8084-574035cffcb3-kube-api-access-pp8l9\" (UniqueName: \"kubernetes.io/projected/bcd0752f-b6ee-4fa6-8084-574035cffcb3-kube-api-access-pp8l9\") pod \"bcd0752f-b6ee-4fa6-8084-574035cffcb3\" (UID: \"bcd0752f-b6ee-4fa6-8084-574035cffcb3\") "
Oct 31 13:54:56.694677 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:56.694634    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd0752f-b6ee-4fa6-8084-574035cffcb3-maas-api-openshift-ca" pod "bcd0752f-b6ee-4fa6-8084-574035cffcb3" (UID: "bcd0752f-b6ee-4fa6-8084-574035cffcb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:54:56.695326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:56.695285    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd0752f-b6ee-4fa6-8084-574035cffcb3-maas-api-openshift-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:54:56.696677 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:56.696650    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd0752f-b6ee-4fa6-8084-574035cffcb3-maas-api-xks-ca" pod "bcd0752f-b6ee-4fa6-8084-574035cffcb3" (UID: "bcd0752f-b6ee-4fa6-8084-574035cffcb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:54:56.697063 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:56.697032    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd0752f-b6ee-4fa6-8084-574035cffcb3-kube-api-access-pp8l9" pod "bcd0752f-b6ee-4fa6-8084-574035cffcb3" (UID: "bcd0752f-b6ee-4fa6-8084-574035cffcb3"). InnerVolumeSpecName "kube-api-access-pp8l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:54:56.841314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:56.817738    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd0752f-b6ee-4fa6-8084-574035cffcb3-maas-api-xks-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:54:56.844940 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:56.844912    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp8l9\" (UniqueName: \"kubernetes.io/projected/bcd0752f-b6ee-4fa6-8084-574035cffcb3-kube-api-access-pp8l9\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:54:57.118570 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.118487    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"]
Oct 31 13:54:57.119463 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.119431    4660 cpu_manager.go:414] "containerMap: removing container" podUID="bcd0752f-b6ee-4fa6-8084-574035cffcb3" containerName="revoke-keys"
Oct 31 13:54:57.119463 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.119459    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd0752f-b6ee-4fa6-8084-574035cffcb3" containerName="revoke-keys"
Oct 31 13:54:57.119674 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.119666    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd0752f-b6ee-4fa6-8084-574035cffcb3" containerName="revoke-keys"
Oct 31 13:54:57.123605 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.123583    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.156783 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.156754    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"]
Oct 31 13:54:57.217631 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.217596    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8a47d6-openshift-default-dockercfg-ss75h\""
Oct 31 13:54:57.254378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.254346    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa53f5f-46bc-443c-a9ad-4382eec12895-istiod-ca-cert\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.254545 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.254411    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-credential-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.254545 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.254516    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlw66\" (UniqueName: \"kubernetes.io/projected/faa53f5f-46bc-443c-a9ad-4382eec12895-kube-api-access-hlw66\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.254545 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.254535    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-token\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.254685 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.254575    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-workload-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.254685 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.254672    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-data\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.254756 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.254730    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-podinfo\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.254845 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.254816    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-workload-certs\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.254952 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.254861    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-envoy\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.356639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.356603    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlw66\" (UniqueName: \"kubernetes.io/projected/faa53f5f-46bc-443c-a9ad-4382eec12895-kube-api-access-hlw66\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.356639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.356641    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-token\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.356889 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.356679    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-workload-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.356889 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.356719    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-data\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.356889 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.356767    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-podinfo\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.356889 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.356804    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-workload-certs\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.356889 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.356832    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-envoy\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.357185 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.356935    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa53f5f-46bc-443c-a9ad-4382eec12895-istiod-ca-cert\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.357185 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.357013    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-credential-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.357288 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.357182    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-workload-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.357288 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.357255    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-data\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.357288 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.357281    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-workload-certs\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.357454 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.357437    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-credential-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.357684 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.357659    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa53f5f-46bc-443c-a9ad-4382eec12895-istiod-ca-cert\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.359317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.359280    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-envoy\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.359428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.359372    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-podinfo\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.371051 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.371029    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-token\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.371166 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.371100    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlw66\" (UniqueName: \"kubernetes.io/projected/faa53f5f-46bc-443c-a9ad-4382eec12895-kube-api-access-hlw66\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.456237 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.456205    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"
Oct 31 13:54:57.456237 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.456221    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn" event={"ID":"bcd0752f-b6ee-4fa6-8084-574035cffcb3","Type":"ContainerDied","Data":"0ae3beeca25fc53fa5db7478680aa7c4361f947ae7cdbbe4965659e51b531b20"}
Oct 31 13:54:57.456471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.456275    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ae3beeca25fc53fa5db7478680aa7c4361f947ae7cdbbe4965659e51b531b20"
Oct 31 13:54:57.457069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.457051    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:57.764817 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:54:57.764783    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa53f5f_46bc_443c_a9ad_4382eec12895.slice/crio-68be1037116bd8da9e9f67b13a8b4131419a5093669ca4d8da76e433d192dbb3 WatchSource:0}: Error finding container 68be1037116bd8da9e9f67b13a8b4131419a5093669ca4d8da76e433d192dbb3: Status 404 returned error can't find the container with id 68be1037116bd8da9e9f67b13a8b4131419a5093669ca4d8da76e433d192dbb3
Oct 31 13:54:57.765132 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.765109    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"]
Oct 31 13:54:57.767747 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.767716    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:54:57.767828 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.767795    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:54:57.767874 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.767826    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:54:57.949156 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.949117    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"]
Oct 31 13:54:57.958583 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.958560    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:57.986353 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:57.985075    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-403dff-openshift-default-dockercfg-plpqv\""
Oct 31 13:54:58.050364 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.043690    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"]
Oct 31 13:54:58.096314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.094840    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5b97ff-dbf0-4a26-835f-702598486d61-istio-podinfo\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.096314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.094909    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jjtj\" (UniqueName: \"kubernetes.io/projected/be5b97ff-dbf0-4a26-835f-702598486d61-kube-api-access-2jjtj\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.096314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.094956    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-istio-data\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.096314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.094988    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-workload-socket\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.096314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.095066    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5b97ff-dbf0-4a26-835f-702598486d61-istio-token\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.096314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.095096    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-workload-certs\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.096314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.095179    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5b97ff-dbf0-4a26-835f-702598486d61-istiod-ca-cert\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.096314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.095211    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-istio-envoy\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.096314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.095415    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-credential-socket\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.207328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.205109    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-istio-data\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.207328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.205169    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-workload-socket\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.207328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.205257    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5b97ff-dbf0-4a26-835f-702598486d61-istio-token\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.207328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.205288    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-workload-certs\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.207328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.205393    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5b97ff-dbf0-4a26-835f-702598486d61-istiod-ca-cert\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.207328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.205423    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-istio-envoy\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.207328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.205610    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-credential-socket\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.207328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.205648    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5b97ff-dbf0-4a26-835f-702598486d61-istio-podinfo\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.207328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.205695    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jjtj\" (UniqueName: \"kubernetes.io/projected/be5b97ff-dbf0-4a26-835f-702598486d61-kube-api-access-2jjtj\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.256459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.252272    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-istio-data\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.256459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.252599    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-workload-socket\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.256459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.253150    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-workload-certs\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.256459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.253887    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5b97ff-dbf0-4a26-835f-702598486d61-istiod-ca-cert\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.318093 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.282072    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-istio-envoy\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.318093 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.282467    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-credential-socket\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.318093 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.316347    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5b97ff-dbf0-4a26-835f-702598486d61-istio-token\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.358427 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.358072    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jjtj\" (UniqueName: \"kubernetes.io/projected/be5b97ff-dbf0-4a26-835f-702598486d61-kube-api-access-2jjtj\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.412862 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.412824    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5b97ff-dbf0-4a26-835f-702598486d61-istio-podinfo\") pod \"e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") " pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.483991 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.483808    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm" event={"ID":"faa53f5f-46bc-443c-a9ad-4382eec12895","Type":"ContainerStarted","Data":"7046d92a75745ce1b5cf75e391f2765608877098d1d12f3907fead0e00c6d741"}
Oct 31 13:54:58.483991 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.483882    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm" event={"ID":"faa53f5f-46bc-443c-a9ad-4382eec12895","Type":"ContainerStarted","Data":"68be1037116bd8da9e9f67b13a8b4131419a5093669ca4d8da76e433d192dbb3"}
Oct 31 13:54:58.630482 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.630451    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:54:58.955558 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.955498    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm" podStartSLOduration=1.9554792669999999 podStartE2EDuration="1.955479267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:58.538640125 +0000 UTC m=+3098.157299291" watchObservedRunningTime="2026-10-31 13:54:58.955479267 +0000 UTC m=+3098.574138433"
Oct 31 13:54:58.957014 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:54:58.956985    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5b97ff_dbf0_4a26_835f_702598486d61.slice/crio-21e79708ca88a752397a9e500d112023675283a1f7a946bacebc7c3482290c5d WatchSource:0}: Error finding container 21e79708ca88a752397a9e500d112023675283a1f7a946bacebc7c3482290c5d: Status 404 returned error can't find the container with id 21e79708ca88a752397a9e500d112023675283a1f7a946bacebc7c3482290c5d
Oct 31 13:54:58.959459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.959432    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"]
Oct 31 13:54:58.960244 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.960211    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:54:58.960323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.960282    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:54:58.960371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:58.960336    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:54:59.457341 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:59.457280    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:54:59.459142 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:59.459114    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:54:59.459288 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:59.459171    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm" podUID="faa53f5f-46bc-443c-a9ad-4382eec12895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 13:54:59.494847 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:59.494802    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" event={"ID":"be5b97ff-dbf0-4a26-835f-702598486d61","Type":"ContainerStarted","Data":"98eea87a15642123c9cb95dada309db6c76f7df967d07c7089babebf9355452b"}
Oct 31 13:54:59.495030 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:59.494868    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" event={"ID":"be5b97ff-dbf0-4a26-835f-702598486d61","Type":"ContainerStarted","Data":"21e79708ca88a752397a9e500d112023675283a1f7a946bacebc7c3482290c5d"}
Oct 31 13:54:59.525004 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:59.524942    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" podStartSLOduration=2.524924748 podStartE2EDuration="2.524924748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:59.51841299 +0000 UTC m=+3099.137072158" watchObservedRunningTime="2026-10-31 13:54:59.524924748 +0000 UTC m=+3099.143583912"
Oct 31 13:54:59.636314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:54:59.635378    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:55:00.459371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.457750    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:55:00.459371 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.457810    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm" podUID="faa53f5f-46bc-443c-a9ad-4382eec12895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 13:55:00.654438 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.654079    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:55:00.661188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.661151    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"]
Oct 31 13:55:00.666142 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.666117    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.672424 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.672398    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a81979-openshift-default-dockercfg-bg46j\""
Oct 31 13:55:00.684147 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.684124    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"]
Oct 31 13:55:00.783394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.777262    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flm7n\" (UniqueName: \"kubernetes.io/projected/cc3f6cec-989a-4415-8328-389d1610947f-kube-api-access-flm7n\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.783394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.777360    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.783394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.777425    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc3f6cec-989a-4415-8328-389d1610947f-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.783394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.777523    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.783394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.777579    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc3f6cec-989a-4415-8328-389d1610947f-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.783394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.777698    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc3f6cec-989a-4415-8328-389d1610947f-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.783394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.777810    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.783394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.777836    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.783394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.777866    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.878956 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.878917    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.878956 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.878959    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.879179 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.879082    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.879179 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.879157    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flm7n\" (UniqueName: \"kubernetes.io/projected/cc3f6cec-989a-4415-8328-389d1610947f-kube-api-access-flm7n\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.879255 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.879194    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.879326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.879265    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc3f6cec-989a-4415-8328-389d1610947f-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.879410 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.879390    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.879672 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.879643    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.879862 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.879834    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.879922 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.879876    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.879922 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.879908    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.879999 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.879971    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc3f6cec-989a-4415-8328-389d1610947f-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.880402 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.880386    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc3f6cec-989a-4415-8328-389d1610947f-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.881168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.881145    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc3f6cec-989a-4415-8328-389d1610947f-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.881965 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.881933    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.882547 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.882508    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc3f6cec-989a-4415-8328-389d1610947f-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.893232 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.893078    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flm7n\" (UniqueName: \"kubernetes.io/projected/cc3f6cec-989a-4415-8328-389d1610947f-kube-api-access-flm7n\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.893232 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.893184    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc3f6cec-989a-4415-8328-389d1610947f-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:00.985537 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:00.985498    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:01.393485 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:01.393451    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:55:01.393631 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:01.393606    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:55:01.393722 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:01.393665    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:55:01.395200 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:01.395176    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"]
Oct 31 13:55:01.458328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:01.458287    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:55:01.458457 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:01.458361    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm" podUID="faa53f5f-46bc-443c-a9ad-4382eec12895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 13:55:01.563537 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:01.563475    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l" event={"ID":"cc3f6cec-989a-4415-8328-389d1610947f","Type":"ContainerStarted","Data":"3d264a2ded8a53a24ea86bccc98aefe7d71d90aa5b291dedc2f65d09fbf6d727"}
Oct 31 13:55:01.564125 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:01.564088    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l" event={"ID":"cc3f6cec-989a-4415-8328-389d1610947f","Type":"ContainerStarted","Data":"6290ed345c073306aac99086ba343fcf1f8db9cb17d5715c281a360d91c2e67e"}
Oct 31 13:55:01.564242 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:01.564230    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:55:01.564426 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:01.564414    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:55:01.592018 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:01.591953    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l" podStartSLOduration=1.5919393689999999 podStartE2EDuration="1.591939369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:01.589739756 +0000 UTC m=+3101.208398923" watchObservedRunningTime="2026-10-31 13:55:01.591939369 +0000 UTC m=+3101.210598533"
Oct 31 13:55:01.986281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:01.986235    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:02.461394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:02.459414    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:55:02.461394 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:02.459478    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm" podUID="faa53f5f-46bc-443c-a9ad-4382eec12895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 13:55:02.987927 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:02.987889    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 13:55:02.988358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:02.987947    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l" podUID="cc3f6cec-989a-4415-8328-389d1610947f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 13:55:03.468323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:03.461371    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:55:03.468323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:03.461435    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm" podUID="faa53f5f-46bc-443c-a9ad-4382eec12895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 13:55:03.987948 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:03.987901    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 13:55:03.988598 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:03.988544    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l" podUID="cc3f6cec-989a-4415-8328-389d1610947f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 13:55:04.458717 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:04.458679    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:55:04.459034 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:04.459002    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm" podUID="faa53f5f-46bc-443c-a9ad-4382eec12895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 13:55:04.986817 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:04.986785    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 13:55:04.986987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:04.986833    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l" podUID="cc3f6cec-989a-4415-8328-389d1610947f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 13:55:05.458193 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:05.458160    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:55:05.458621 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:05.458210    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm" podUID="faa53f5f-46bc-443c-a9ad-4382eec12895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 13:55:05.986187 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:05.986144    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 13:55:05.986400 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:05.986210    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l" podUID="cc3f6cec-989a-4415-8328-389d1610947f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 13:55:06.462767 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:06.462734    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:55:06.463218 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:06.463203    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:55:06.463844 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:06.463821    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:55:06.986332 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:06.986283    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 13:55:06.986516 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:06.986352    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l" podUID="cc3f6cec-989a-4415-8328-389d1610947f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 13:55:07.989694 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:07.989655    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 13:55:07.991024 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:07.990981    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l" podUID="cc3f6cec-989a-4415-8328-389d1610947f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 13:55:08.986018 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:08.985983    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 13:55:08.986251 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:08.986055    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l" podUID="cc3f6cec-989a-4415-8328-389d1610947f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 13:55:09.994723 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:09.994689    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:09.996179 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:09.996137    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:10.000694 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:10.000664    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 13:55:18.412777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:18.412733    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"]
Oct 31 13:55:20.734235 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:55:20.734193    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:55:22.677052 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:22.677011    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"]
Oct 31 13:55:22.683886 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:22.682935    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:55:22.690658 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:22.690628    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-dafbce-serving-cert\""
Oct 31 13:55:22.710991 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:22.710571    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"]
Oct 31 13:55:22.784931 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:22.784890    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec915fa-dbf8-43f5-bb74-db141931047c-metrics-certs\") pod \"maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg\" (UID: \"cec915fa-dbf8-43f5-bb74-db141931047c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:55:22.785136 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:22.785049    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cec915fa-dbf8-43f5-bb74-db141931047c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg\" (UID: \"cec915fa-dbf8-43f5-bb74-db141931047c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:55:22.785240 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:22.785212    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cec915fa-dbf8-43f5-bb74-db141931047c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg\" (UID: \"cec915fa-dbf8-43f5-bb74-db141931047c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:55:22.785315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:22.785260    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9q2b\" (UniqueName: \"kubernetes.io/projected/cec915fa-dbf8-43f5-bb74-db141931047c-kube-api-access-v9q2b\") pod \"maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg\" (UID: \"cec915fa-dbf8-43f5-bb74-db141931047c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:55:22.887465 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:22.887415    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cec915fa-dbf8-43f5-bb74-db141931047c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg\" (UID: \"cec915fa-dbf8-43f5-bb74-db141931047c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:55:22.887663 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:22.887468    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9q2b\" (UniqueName: \"kubernetes.io/projected/cec915fa-dbf8-43f5-bb74-db141931047c-kube-api-access-v9q2b\") pod \"maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg\" (UID: \"cec915fa-dbf8-43f5-bb74-db141931047c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:55:22.887739 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:22.887704    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec915fa-dbf8-43f5-bb74-db141931047c-metrics-certs\") pod \"maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg\" (UID: \"cec915fa-dbf8-43f5-bb74-db141931047c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:55:22.887833 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:22.887816    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cec915fa-dbf8-43f5-bb74-db141931047c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg\" (UID: \"cec915fa-dbf8-43f5-bb74-db141931047c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:55:22.888507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:22.888477    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cec915fa-dbf8-43f5-bb74-db141931047c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg\" (UID: \"cec915fa-dbf8-43f5-bb74-db141931047c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:55:22.890546 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:22.890515    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec915fa-dbf8-43f5-bb74-db141931047c-metrics-certs\") pod \"maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg\" (UID: \"cec915fa-dbf8-43f5-bb74-db141931047c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:55:22.890664 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:22.890597    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cec915fa-dbf8-43f5-bb74-db141931047c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg\" (UID: \"cec915fa-dbf8-43f5-bb74-db141931047c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:55:22.900211 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:22.900188    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9q2b\" (UniqueName: \"kubernetes.io/projected/cec915fa-dbf8-43f5-bb74-db141931047c-kube-api-access-v9q2b\") pod \"maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg\" (UID: \"cec915fa-dbf8-43f5-bb74-db141931047c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:55:23.038448 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:23.038358    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:55:23.487974 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:23.487940    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"]
Oct 31 13:55:23.507344 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:23.507263    4660 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:55:23.926493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:23.925434    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg" event={"ID":"cec915fa-dbf8-43f5-bb74-db141931047c","Type":"ContainerStarted","Data":"c06ecba01373bae0f04196c965eaca1180561e02a62484700cb0798020925df9"}
Oct 31 13:55:24.659570 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.659533    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"]
Oct 31 13:55:24.664181 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.664156    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.671630 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.671607    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-599291-openshift-default-dockercfg-9db8q\""
Oct 31 13:55:24.677734 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.677711    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"]
Oct 31 13:55:24.773319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.768439    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-workload-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.773319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.768630    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-workload-certs\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.773319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.768683    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-token\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.773319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.768743    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-credential-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.773319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.768920    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b061377-3c76-4eb8-aa22-04425a306fe6-istiod-ca-cert\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.773319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.768964    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-envoy\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.773319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.769034    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-data\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.773319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.769263    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrwfb\" (UniqueName: \"kubernetes.io/projected/1b061377-3c76-4eb8-aa22-04425a306fe6-kube-api-access-wrwfb\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.773319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.769326    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-podinfo\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.870611 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.870572    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-workload-certs\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.870611 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.870614    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-token\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.870860 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.870654    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-credential-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.870860 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.870766    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b061377-3c76-4eb8-aa22-04425a306fe6-istiod-ca-cert\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.870860 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.870797    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-envoy\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.870860 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.870846    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-data\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.871057 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.871030    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-workload-certs\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.871057 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.871046    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrwfb\" (UniqueName: \"kubernetes.io/projected/1b061377-3c76-4eb8-aa22-04425a306fe6-kube-api-access-wrwfb\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.871132 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.871093    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-podinfo\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.871132 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.871114    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-credential-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.871202 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.871144    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-workload-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.871461 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.871430    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-data\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.871601 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.871579    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b061377-3c76-4eb8-aa22-04425a306fe6-istiod-ca-cert\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.871656 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.871579    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-workload-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.873157 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.873138    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-envoy\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.873699 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.873671    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-podinfo\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.884064 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.884042    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-token\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.884681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.884125    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrwfb\" (UniqueName: \"kubernetes.io/projected/1b061377-3c76-4eb8-aa22-04425a306fe6-kube-api-access-wrwfb\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:24.935367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.935256    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg" event={"ID":"cec915fa-dbf8-43f5-bb74-db141931047c","Type":"ContainerStarted","Data":"63d743d66f41a33257ceacb41a9195e20cba6da4d2cd34f1521dd626504e396f"}
Oct 31 13:55:24.935751 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.935456    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:55:24.957287 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.957239    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg" podStartSLOduration=2.388704777 podStartE2EDuration="2.957226549s" totalImagesPullingTime="568.521772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:24.956580056 +0000 UTC m=+3124.575239245" watchObservedRunningTime="2026-10-31 13:55:24.957226549 +0000 UTC m=+3124.575885712"
Oct 31 13:55:24.982962 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:24.982936    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:25.282276 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:55:25.282247    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b061377_3c76_4eb8_aa22_04425a306fe6.slice/crio-ec32cd02b199bf4273851aa10160b7740f26d33c6269256f438dc2c402f5cd46 WatchSource:0}: Error finding container ec32cd02b199bf4273851aa10160b7740f26d33c6269256f438dc2c402f5cd46: Status 404 returned error can't find the container with id ec32cd02b199bf4273851aa10160b7740f26d33c6269256f438dc2c402f5cd46
Oct 31 13:55:25.282881 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:25.282857    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"]
Oct 31 13:55:25.285857 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:25.285824    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:55:25.285947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:25.285891    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:55:25.285947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:25.285939    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:55:25.952903 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:25.952860    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47" event={"ID":"1b061377-3c76-4eb8-aa22-04425a306fe6","Type":"ContainerStarted","Data":"5939af1968b7653c731e7fae17ca0019275996f6cf9f381c9745e358c261823d"}
Oct 31 13:55:25.954575 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:25.952926    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47" event={"ID":"1b061377-3c76-4eb8-aa22-04425a306fe6","Type":"ContainerStarted","Data":"ec32cd02b199bf4273851aa10160b7740f26d33c6269256f438dc2c402f5cd46"}
Oct 31 13:55:25.983414 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:25.983379    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:25.986223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:25.986171    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47" podStartSLOduration=1.986157141 podStartE2EDuration="1.986157141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:25.980208202 +0000 UTC m=+3125.598867375" watchObservedRunningTime="2026-10-31 13:55:25.986157141 +0000 UTC m=+3125.604816304"
Oct 31 13:55:26.985585 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:26.985550    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 13:55:26.986048 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:26.985609    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47" podUID="1b061377-3c76-4eb8-aa22-04425a306fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 13:55:27.984425 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:27.984159    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 13:55:27.984425 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:27.984211    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47" podUID="1b061377-3c76-4eb8-aa22-04425a306fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 13:55:28.255376 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.252053    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"]
Oct 31 13:55:28.263342 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.261860    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 13:55:28.267789 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.267456    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"]
Oct 31 13:55:28.274918 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.271682    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-403dff-serving-cert\""
Oct 31 13:55:28.441367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.436435    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f3684e-26dd-493c-af92-89e1f8b2b18a-metrics-certs\") pod \"maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn\" (UID: \"21f3684e-26dd-493c-af92-89e1f8b2b18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 13:55:28.441367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.436517    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxv94\" (UniqueName: \"kubernetes.io/projected/21f3684e-26dd-493c-af92-89e1f8b2b18a-kube-api-access-wxv94\") pod \"maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn\" (UID: \"21f3684e-26dd-493c-af92-89e1f8b2b18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 13:55:28.441367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.436550    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f3684e-26dd-493c-af92-89e1f8b2b18a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn\" (UID: \"21f3684e-26dd-493c-af92-89e1f8b2b18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 13:55:28.441367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.436597    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f3684e-26dd-493c-af92-89e1f8b2b18a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn\" (UID: \"21f3684e-26dd-493c-af92-89e1f8b2b18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 13:55:28.542380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.541401    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f3684e-26dd-493c-af92-89e1f8b2b18a-metrics-certs\") pod \"maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn\" (UID: \"21f3684e-26dd-493c-af92-89e1f8b2b18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 13:55:28.542380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.541474    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxv94\" (UniqueName: \"kubernetes.io/projected/21f3684e-26dd-493c-af92-89e1f8b2b18a-kube-api-access-wxv94\") pod \"maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn\" (UID: \"21f3684e-26dd-493c-af92-89e1f8b2b18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 13:55:28.542380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.541507    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f3684e-26dd-493c-af92-89e1f8b2b18a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn\" (UID: \"21f3684e-26dd-493c-af92-89e1f8b2b18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 13:55:28.542380 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.541561    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f3684e-26dd-493c-af92-89e1f8b2b18a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn\" (UID: \"21f3684e-26dd-493c-af92-89e1f8b2b18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 13:55:28.546320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.543760    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f3684e-26dd-493c-af92-89e1f8b2b18a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn\" (UID: \"21f3684e-26dd-493c-af92-89e1f8b2b18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 13:55:28.572315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.571972    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxv94\" (UniqueName: \"kubernetes.io/projected/21f3684e-26dd-493c-af92-89e1f8b2b18a-kube-api-access-wxv94\") pod \"maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn\" (UID: \"21f3684e-26dd-493c-af92-89e1f8b2b18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 13:55:28.572498 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.572484    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f3684e-26dd-493c-af92-89e1f8b2b18a-metrics-certs\") pod \"maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn\" (UID: \"21f3684e-26dd-493c-af92-89e1f8b2b18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 13:55:28.581313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.579894    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f3684e-26dd-493c-af92-89e1f8b2b18a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn\" (UID: \"21f3684e-26dd-493c-af92-89e1f8b2b18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 13:55:28.652575 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.652539    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 13:55:28.994314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.985196    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 13:55:28.994314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:28.985258    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47" podUID="1b061377-3c76-4eb8-aa22-04425a306fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 13:55:29.498383 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:29.498351    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"]
Oct 31 13:55:29.501094 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:55:29.501064    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f3684e_26dd_493c_af92_89e1f8b2b18a.slice/crio-2d4e5b803a02d06ac34cede84d2ec4cb8ab95c99368c258712a3fb2ed4c96a61 WatchSource:0}: Error finding container 2d4e5b803a02d06ac34cede84d2ec4cb8ab95c99368c258712a3fb2ed4c96a61: Status 404 returned error can't find the container with id 2d4e5b803a02d06ac34cede84d2ec4cb8ab95c99368c258712a3fb2ed4c96a61
Oct 31 13:55:29.985166 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:29.984518    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 13:55:29.985166 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:29.984583    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47" podUID="1b061377-3c76-4eb8-aa22-04425a306fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 13:55:30.167244 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:30.167202    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn" event={"ID":"21f3684e-26dd-493c-af92-89e1f8b2b18a","Type":"ContainerStarted","Data":"2d4e5b803a02d06ac34cede84d2ec4cb8ab95c99368c258712a3fb2ed4c96a61"}
Oct 31 13:55:30.880355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:30.879311    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"]
Oct 31 13:55:30.977651 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:30.977613    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:55:30.985352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:30.983661    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 13:55:30.985352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:30.983714    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47" podUID="1b061377-3c76-4eb8-aa22-04425a306fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 13:55:31.330401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:31.325819    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn" event={"ID":"21f3684e-26dd-493c-af92-89e1f8b2b18a","Type":"ContainerStarted","Data":"b4ab240a68ebc8c7417c999d991182d4c010ee9cd190d54b303ba568fad18fd2"}
Oct 31 13:55:31.330401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:31.326097    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 13:55:31.370334 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:31.368106    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn" podStartSLOduration=2.80235975 podStartE2EDuration="3.36808467s" totalImagesPullingTime="565.72492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:31.365004529 +0000 UTC m=+3130.983663695" watchObservedRunningTime="2026-10-31 13:55:31.36808467 +0000 UTC m=+3130.986743835"
Oct 31 13:55:31.986605 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:31.985713    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 13:55:31.986605 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:31.985789    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47" podUID="1b061377-3c76-4eb8-aa22-04425a306fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 13:55:32.988902 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:32.988870    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:32.989436 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:32.989416    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:32.990184 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:32.990165    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 13:55:33.458684 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:33.458637    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f" podUID="36787aaf-9893-4940-b801-c90aa192d629" containerName="main" containerID="cri-o://47e4a5575e0e9cd37aafb32b0f8ce882b3be761c59cc6bcf9a9419ceb357f3fd" gracePeriod=45
Oct 31 13:55:33.487828 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:33.487797    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f" podUID="36787aaf-9893-4940-b801-c90aa192d629" containerName="main" probeResult="failure" output="Get \"https://10.133.0.166:8000/ready\": dial tcp 10.133.0.166:8000: connect: connection refused"
Oct 31 13:55:33.860372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:33.860346    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:55:34.042158 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.042125    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-dshm\") pod \"36787aaf-9893-4940-b801-c90aa192d629\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") "
Oct 31 13:55:34.042680 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.042235    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36787aaf-9893-4940-b801-c90aa192d629-tls-certs\" (UniqueName: \"kubernetes.io/secret/36787aaf-9893-4940-b801-c90aa192d629-tls-certs\") pod \"36787aaf-9893-4940-b801-c90aa192d629\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") "
Oct 31 13:55:34.042680 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.042264    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36787aaf-9893-4940-b801-c90aa192d629-kube-api-access-cjr8h\" (UniqueName: \"kubernetes.io/projected/36787aaf-9893-4940-b801-c90aa192d629-kube-api-access-cjr8h\") pod \"36787aaf-9893-4940-b801-c90aa192d629\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") "
Oct 31 13:55:34.042680 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.042315    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-home\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-home\") pod \"36787aaf-9893-4940-b801-c90aa192d629\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") "
Oct 31 13:55:34.042680 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.042404    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-tmp-dir\") pod \"36787aaf-9893-4940-b801-c90aa192d629\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") "
Oct 31 13:55:34.042680 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.042444    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-model-cache\") pod \"36787aaf-9893-4940-b801-c90aa192d629\" (UID: \"36787aaf-9893-4940-b801-c90aa192d629\") "
Oct 31 13:55:34.042960 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.042765    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-home" pod "36787aaf-9893-4940-b801-c90aa192d629" (UID: "36787aaf-9893-4940-b801-c90aa192d629"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:34.043021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.042959    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-tmp-dir" pod "36787aaf-9893-4940-b801-c90aa192d629" (UID: "36787aaf-9893-4940-b801-c90aa192d629"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:34.043021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.042973    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-model-cache" pod "36787aaf-9893-4940-b801-c90aa192d629" (UID: "36787aaf-9893-4940-b801-c90aa192d629"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:34.043187 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.043170    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:34.043255 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.043188    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:34.043255 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.043202    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:34.046053 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.045088    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36787aaf-9893-4940-b801-c90aa192d629-tls-certs" pod "36787aaf-9893-4940-b801-c90aa192d629" (UID: "36787aaf-9893-4940-b801-c90aa192d629"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:55:34.046053 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.045245    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-dshm" pod "36787aaf-9893-4940-b801-c90aa192d629" (UID: "36787aaf-9893-4940-b801-c90aa192d629"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:34.046678 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.046650    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36787aaf-9893-4940-b801-c90aa192d629-kube-api-access-cjr8h" pod "36787aaf-9893-4940-b801-c90aa192d629" (UID: "36787aaf-9893-4940-b801-c90aa192d629"). InnerVolumeSpecName "kube-api-access-cjr8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:55:34.145375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.145277    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36787aaf-9893-4940-b801-c90aa192d629-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:34.145375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.145339    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjr8h\" (UniqueName: \"kubernetes.io/projected/36787aaf-9893-4940-b801-c90aa192d629-kube-api-access-cjr8h\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:34.145375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.145354    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36787aaf-9893-4940-b801-c90aa192d629-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:34.363695 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.363652    4660 generic.go:356] "Generic (PLEG): container finished" podID="36787aaf-9893-4940-b801-c90aa192d629" containerID="47e4a5575e0e9cd37aafb32b0f8ce882b3be761c59cc6bcf9a9419ceb357f3fd" exitCode=0
Oct 31 13:55:34.365078 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.365050    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"
Oct 31 13:55:34.365078 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.365064    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f" event={"ID":"36787aaf-9893-4940-b801-c90aa192d629","Type":"ContainerDied","Data":"47e4a5575e0e9cd37aafb32b0f8ce882b3be761c59cc6bcf9a9419ceb357f3fd"}
Oct 31 13:55:34.365287 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.365114    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f" event={"ID":"36787aaf-9893-4940-b801-c90aa192d629","Type":"ContainerDied","Data":"139920ff3c34dcf0a4d827104eb16ebe5a4bdff6e0afad4a63e762e7d07c824e"}
Oct 31 13:55:34.365287 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.365138    4660 scope.go:122] "RemoveContainer" containerID="47e4a5575e0e9cd37aafb32b0f8ce882b3be761c59cc6bcf9a9419ceb357f3fd"
Oct 31 13:55:34.382647 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.382624    4660 scope.go:122] "RemoveContainer" containerID="47e4a5575e0e9cd37aafb32b0f8ce882b3be761c59cc6bcf9a9419ceb357f3fd"
Oct 31 13:55:34.382976 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:55:34.382948    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e4a5575e0e9cd37aafb32b0f8ce882b3be761c59cc6bcf9a9419ceb357f3fd\": container with ID starting with 47e4a5575e0e9cd37aafb32b0f8ce882b3be761c59cc6bcf9a9419ceb357f3fd not found: ID does not exist" containerID="47e4a5575e0e9cd37aafb32b0f8ce882b3be761c59cc6bcf9a9419ceb357f3fd"
Oct 31 13:55:34.383073 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.382988    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e4a5575e0e9cd37aafb32b0f8ce882b3be761c59cc6bcf9a9419ceb357f3fd"} err="failed to get container status \"47e4a5575e0e9cd37aafb32b0f8ce882b3be761c59cc6bcf9a9419ceb357f3fd\": rpc error: code = NotFound desc = could not find container \"47e4a5575e0e9cd37aafb32b0f8ce882b3be761c59cc6bcf9a9419ceb357f3fd\": container with ID starting with 47e4a5575e0e9cd37aafb32b0f8ce882b3be761c59cc6bcf9a9419ceb357f3fd not found: ID does not exist"
Oct 31 13:55:34.396582 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.396519    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"]
Oct 31 13:55:34.403846 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.403816    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a39e7e/auth-test-model-a39e7e-kserve-5b8c5845bf-xc47f"]
Oct 31 13:55:34.543557 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:34.543510    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36787aaf-9893-4940-b801-c90aa192d629" path="/var/lib/kubelet/pods/36787aaf-9893-4940-b801-c90aa192d629/volumes"
Oct 31 13:55:37.344104 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:37.344075    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 13:55:45.761119 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.761076    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"]
Oct 31 13:55:45.762801 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.762771    4660 cpu_manager.go:414] "containerMap: removing container" podUID="36787aaf-9893-4940-b801-c90aa192d629" containerName="main"
Oct 31 13:55:45.762983 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.762956    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36787aaf-9893-4940-b801-c90aa192d629" containerName="main"
Oct 31 13:55:45.763178 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.763162    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="36787aaf-9893-4940-b801-c90aa192d629" containerName="main"
Oct 31 13:55:45.772308 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.770184    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.781103 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.780431    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"]
Oct 31 13:55:45.884975 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.884930    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.885179 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.884984    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.885179 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.885031    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.885179 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.885107    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nm59\" (UniqueName: \"kubernetes.io/projected/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-kube-api-access-6nm59\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.885179 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.885149    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.885466 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.885215    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.885466 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.885254    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.885466 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.885286    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.885466 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.885421    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.974605 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.974567    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r" podUID="f682ebee-c7de-42ae-a1d1-3242a0a2f0e7" containerName="main" containerID="cri-o://c8d456b3288f73ecdf12f8eabf05e617bb0254ac7dac81ecfed42237fa870e9d" gracePeriod=45
Oct 31 13:55:45.986947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.986914    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.987052 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.986998    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nm59\" (UniqueName: \"kubernetes.io/projected/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-kube-api-access-6nm59\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.987052 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.987039    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.987169 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.987096    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.987169 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.987131    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.987169 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.987161    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.987446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.987265    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.987446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.987313    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.987446 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.987347    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.987618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.987571    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.987676 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.987612    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.987676 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.987648    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.987824 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.987799    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.987952 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.987922    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.989855 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.989828    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.990121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.990105    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:45.999971 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:45.999947    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:46.000209 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.000190    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nm59\" (UniqueName: \"kubernetes.io/projected/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-kube-api-access-6nm59\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:46.134424 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.134387    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:46.303023 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.302989    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:55:46.393317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.392286    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-dshm\") pod \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") "
Oct 31 13:55:46.393317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.392541    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-tls-certs\") pod \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") "
Oct 31 13:55:46.393317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.392577    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-model-cache\") pod \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") "
Oct 31 13:55:46.393317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.392640    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-home\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-home\") pod \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") "
Oct 31 13:55:46.393317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.392696    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-tmp-dir\") pod \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") "
Oct 31 13:55:46.393317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.392832    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-kube-api-access-4nqr2\" (UniqueName: \"kubernetes.io/projected/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-kube-api-access-4nqr2\") pod \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\" (UID: \"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7\") "
Oct 31 13:55:46.396314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.394493    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-model-cache" pod "f682ebee-c7de-42ae-a1d1-3242a0a2f0e7" (UID: "f682ebee-c7de-42ae-a1d1-3242a0a2f0e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:46.396837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.396771    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-home" pod "f682ebee-c7de-42ae-a1d1-3242a0a2f0e7" (UID: "f682ebee-c7de-42ae-a1d1-3242a0a2f0e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:46.396837 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.396795    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-tmp-dir" pod "f682ebee-c7de-42ae-a1d1-3242a0a2f0e7" (UID: "f682ebee-c7de-42ae-a1d1-3242a0a2f0e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:46.397003 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.396973    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-kube-api-access-4nqr2" pod "f682ebee-c7de-42ae-a1d1-3242a0a2f0e7" (UID: "f682ebee-c7de-42ae-a1d1-3242a0a2f0e7"). InnerVolumeSpecName "kube-api-access-4nqr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:55:46.400320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.399385    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-tls-certs" pod "f682ebee-c7de-42ae-a1d1-3242a0a2f0e7" (UID: "f682ebee-c7de-42ae-a1d1-3242a0a2f0e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:55:46.411349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.405908    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-dshm" pod "f682ebee-c7de-42ae-a1d1-3242a0a2f0e7" (UID: "f682ebee-c7de-42ae-a1d1-3242a0a2f0e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:46.469553 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.469522    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"]
Oct 31 13:55:46.470011 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:55:46.469988    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2dc5c7e_d033_4f1d_92e9_b8c57128d7a8.slice/crio-d15cfec7798e0f8d4214e81d3584aeb16d095e2af774ac6a0918bb55e14429eb WatchSource:0}: Error finding container d15cfec7798e0f8d4214e81d3584aeb16d095e2af774ac6a0918bb55e14429eb: Status 404 returned error can't find the container with id d15cfec7798e0f8d4214e81d3584aeb16d095e2af774ac6a0918bb55e14429eb
Oct 31 13:55:46.473370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.473330    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:55:46.473470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.473414    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:55:46.473470 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.473455    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:55:46.495512 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.495485    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:46.495618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.495517    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nqr2\" (UniqueName: \"kubernetes.io/projected/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-kube-api-access-4nqr2\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:46.495618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.495534    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:46.495618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.495549    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:46.495618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.495565    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:46.495618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.495578    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:46.506090 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.506060    4660 generic.go:356] "Generic (PLEG): container finished" podID="f682ebee-c7de-42ae-a1d1-3242a0a2f0e7" containerID="c8d456b3288f73ecdf12f8eabf05e617bb0254ac7dac81ecfed42237fa870e9d" exitCode=0
Oct 31 13:55:46.506201 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.506113    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"
Oct 31 13:55:46.506201 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.506139    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r" event={"ID":"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7","Type":"ContainerDied","Data":"c8d456b3288f73ecdf12f8eabf05e617bb0254ac7dac81ecfed42237fa870e9d"}
Oct 31 13:55:46.506201 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.506186    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r" event={"ID":"f682ebee-c7de-42ae-a1d1-3242a0a2f0e7","Type":"ContainerDied","Data":"55c7f67264ef4be8da81435547ce14e300dfa3e91f60160336099cfdd64990b1"}
Oct 31 13:55:46.506374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.506207    4660 scope.go:122] "RemoveContainer" containerID="c8d456b3288f73ecdf12f8eabf05e617bb0254ac7dac81ecfed42237fa870e9d"
Oct 31 13:55:46.508061 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.508036    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk" event={"ID":"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8","Type":"ContainerStarted","Data":"d15cfec7798e0f8d4214e81d3584aeb16d095e2af774ac6a0918bb55e14429eb"}
Oct 31 13:55:46.529832 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.525429    4660 scope.go:122] "RemoveContainer" containerID="c8d456b3288f73ecdf12f8eabf05e617bb0254ac7dac81ecfed42237fa870e9d"
Oct 31 13:55:46.529832 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:55:46.526387    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d456b3288f73ecdf12f8eabf05e617bb0254ac7dac81ecfed42237fa870e9d\": container with ID starting with c8d456b3288f73ecdf12f8eabf05e617bb0254ac7dac81ecfed42237fa870e9d not found: ID does not exist" containerID="c8d456b3288f73ecdf12f8eabf05e617bb0254ac7dac81ecfed42237fa870e9d"
Oct 31 13:55:46.529832 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.526421    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d456b3288f73ecdf12f8eabf05e617bb0254ac7dac81ecfed42237fa870e9d"} err="failed to get container status \"c8d456b3288f73ecdf12f8eabf05e617bb0254ac7dac81ecfed42237fa870e9d\": rpc error: code = NotFound desc = could not find container \"c8d456b3288f73ecdf12f8eabf05e617bb0254ac7dac81ecfed42237fa870e9d\": container with ID starting with c8d456b3288f73ecdf12f8eabf05e617bb0254ac7dac81ecfed42237fa870e9d not found: ID does not exist"
Oct 31 13:55:46.549674 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.549623    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"]
Oct 31 13:55:46.556853 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:46.556828    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-974711/auth-test-model-974711-kserve-8d45f44f-hzq8r"]
Oct 31 13:55:47.561503 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:47.561462    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk" event={"ID":"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8","Type":"ContainerStarted","Data":"4327c67ab5b09bd35e1f12e27ad8352a828b51155b2fb447bba2cc062ca047c2"}
Oct 31 13:55:47.590487 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:47.590424    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk" podStartSLOduration=2.590406849 podStartE2EDuration="2.590406849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:47.588349509 +0000 UTC m=+3147.207008674" watchObservedRunningTime="2026-10-31 13:55:47.590406849 +0000 UTC m=+3147.209066017"
Oct 31 13:55:48.138333 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:48.137644    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:48.151339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:48.150426    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 13:55:48.151339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:48.150493    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk" podUID="f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 13:55:48.548797 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:48.548712    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f682ebee-c7de-42ae-a1d1-3242a0a2f0e7" path="/var/lib/kubelet/pods/f682ebee-c7de-42ae-a1d1-3242a0a2f0e7/volumes"
Oct 31 13:55:49.135665 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:49.135627    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 13:55:49.136282 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:49.136240    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk" podUID="f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 13:55:50.139317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:50.138660    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 13:55:50.139317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:50.138731    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk" podUID="f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 13:55:50.878318 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:55:50.864643    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:55:51.140641 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:51.140552    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 13:55:51.141187 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:51.141154    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk" podUID="f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 13:55:52.135735 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:52.135689    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 13:55:52.136015 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:52.135986    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk" podUID="f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 13:55:53.136249 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:53.135657    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 13:55:53.136249 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:53.135722    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk" podUID="f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 13:55:54.149646 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:54.149610    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:54.152681 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:54.152650    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:55:54.160330 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:55:54.157565    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 13:56:00.241763 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.236719    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"]
Oct 31 13:56:00.761253 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.761213    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"]
Oct 31 13:56:00.762500 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.762472    4660 cpu_manager.go:414] "containerMap: removing container" podUID="f682ebee-c7de-42ae-a1d1-3242a0a2f0e7" containerName="main"
Oct 31 13:56:00.762647 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.762634    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f682ebee-c7de-42ae-a1d1-3242a0a2f0e7" containerName="main"
Oct 31 13:56:00.762911 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.762899    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="f682ebee-c7de-42ae-a1d1-3242a0a2f0e7" containerName="main"
Oct 31 13:56:00.779068 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.779031    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.780186 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.780155    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"]
Oct 31 13:56:00.868951 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.868829    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146bb004-162d-4777-92b9-08b3120df4d1-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.869281 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.869240    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6768n\" (UniqueName: \"kubernetes.io/projected/146bb004-162d-4777-92b9-08b3120df4d1-kube-api-access-6768n\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.869597 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.869576    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.869819 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.869800    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146bb004-162d-4777-92b9-08b3120df4d1-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.870074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.870053    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146bb004-162d-4777-92b9-08b3120df4d1-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.879187 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.879158    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.879520 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.879497    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.879873 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.879850    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.889943 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.889916    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.991279 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.991242    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.991279 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.991280    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.991548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.991380    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146bb004-162d-4777-92b9-08b3120df4d1-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.991548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.991428    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6768n\" (UniqueName: \"kubernetes.io/projected/146bb004-162d-4777-92b9-08b3120df4d1-kube-api-access-6768n\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.991548 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.991486    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.994705 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.994658    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.994839 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.994733    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146bb004-162d-4777-92b9-08b3120df4d1-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.994898 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.994866    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146bb004-162d-4777-92b9-08b3120df4d1-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.994954 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.994926    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.994954 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.994941    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.995059 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.995012    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.995238 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.995219    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.995484 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.995462    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:00.996002 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:00.995976    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146bb004-162d-4777-92b9-08b3120df4d1-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:01.007906 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:01.007871    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146bb004-162d-4777-92b9-08b3120df4d1-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:01.011760 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:01.011677    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6768n\" (UniqueName: \"kubernetes.io/projected/146bb004-162d-4777-92b9-08b3120df4d1-kube-api-access-6768n\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:01.014584 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:01.014532    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146bb004-162d-4777-92b9-08b3120df4d1-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:01.017835 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:01.017789    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:01.104965 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:01.104919    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:01.499370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:01.499335    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"]
Oct 31 13:56:01.508526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:01.508478    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:56:01.508662 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:01.508579    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:56:01.508726 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:01.508660    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:56:02.221054 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:02.221019    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp" event={"ID":"146bb004-162d-4777-92b9-08b3120df4d1","Type":"ContainerStarted","Data":"95cbdaf62050d358226fef728a8f495a5bc76f63669868fcf2b6b7d4b381639e"}
Oct 31 13:56:02.221398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:02.221375    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp" event={"ID":"146bb004-162d-4777-92b9-08b3120df4d1","Type":"ContainerStarted","Data":"b4bfcef3f586c49a700feeb0a755b46b06decbc183eaf0fd9072625a274e332f"}
Oct 31 13:56:02.251021 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:02.250957    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp" podStartSLOduration=2.250937303 podStartE2EDuration="2.250937303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:56:02.247950591 +0000 UTC m=+3161.866609756" watchObservedRunningTime="2026-10-31 13:56:02.250937303 +0000 UTC m=+3161.869596468"
Oct 31 13:56:02.604530 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:02.604432    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:56:02.604995 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:02.604517    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:56:03.107568 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:03.107533    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:03.128074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:03.126313    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 31 13:56:03.128074 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:03.126380    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp" podUID="146bb004-162d-4777-92b9-08b3120df4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 31 13:56:04.117324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:04.113009    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 31 13:56:04.117324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:04.113075    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp" podUID="146bb004-162d-4777-92b9-08b3120df4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 31 13:56:05.111751 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:05.111708    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 31 13:56:05.112064 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:05.112033    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp" podUID="146bb004-162d-4777-92b9-08b3120df4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 31 13:56:06.133317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:06.123971    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 31 13:56:06.133317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:06.124031    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp" podUID="146bb004-162d-4777-92b9-08b3120df4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 31 13:56:06.286550 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:06.286469    4660 generic.go:356] "Generic (PLEG): container finished" podID="be5b97ff-dbf0-4a26-835f-702598486d61" containerID="98eea87a15642123c9cb95dada309db6c76f7df967d07c7089babebf9355452b" exitCode=0
Oct 31 13:56:06.286702 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:06.286542    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" event={"ID":"be5b97ff-dbf0-4a26-835f-702598486d61","Type":"ContainerDied","Data":"98eea87a15642123c9cb95dada309db6c76f7df967d07c7089babebf9355452b"}
Oct 31 13:56:06.286981 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:06.286967    4660 scope.go:122] "RemoveContainer" containerID="98eea87a15642123c9cb95dada309db6c76f7df967d07c7089babebf9355452b"
Oct 31 13:56:06.288844 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:06.288814    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:56:06.288923 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:06.288887    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:56:06.288961 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:06.288920    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:56:07.109318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:07.105644    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 31 13:56:07.109318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:07.105708    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp" podUID="146bb004-162d-4777-92b9-08b3120df4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 31 13:56:07.296342 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:07.296304    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" event={"ID":"be5b97ff-dbf0-4a26-835f-702598486d61","Type":"ContainerStarted","Data":"85cbf17b3968a7dec0fbd0f1d08c0d8c9de8017188454e241c5a3a5a507540ab"}
Oct 31 13:56:07.671399 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:07.664341    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:56:07.671399 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:07.664518    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:56:07.671399 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:07.664577    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 31 13:56:08.108372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:08.105743    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 31 13:56:08.108372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:08.105814    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp" podUID="146bb004-162d-4777-92b9-08b3120df4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 31 13:56:08.648309 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:08.634786    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:56:08.648309 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:08.634844    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 31 13:56:09.124400 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:09.123437    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 31 13:56:09.124400 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:09.123498    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp" podUID="146bb004-162d-4777-92b9-08b3120df4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 31 13:56:09.638940 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:09.638878    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:56:09.639220 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:09.639187    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 31 13:56:09.693777 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:09.693718    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"]
Oct 31 13:56:10.154417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:10.154374    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:10.219321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:10.212135    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:10.224586 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:10.224051    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 13:56:10.636317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:10.634698    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:56:10.636317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:10.634783    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 31 13:56:11.635319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:11.634612    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:56:11.635319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:11.634674    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 31 13:56:12.630887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:12.630852    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:56:12.631166 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:12.631137    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 31 13:56:13.644382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:13.631960    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:56:13.644382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:13.632031    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 31 13:56:14.635669 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:14.635623    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:56:14.636192 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:14.636165    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:56:14.636866 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:14.636844    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 13:56:15.262644 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.262593    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m" podUID="7c10f8e2-7ba7-4191-bb14-6a09b346f3a8" containerName="main" containerID="cri-o://e4017acfc3d16e26a78ff0c47cc41922aeb5c59c0a7dfb499f6620d031ebbe4f" gracePeriod=45
Oct 31 13:56:15.419599 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.419559    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m" podUID="7c10f8e2-7ba7-4191-bb14-6a09b346f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.169:8000/ready\": dial tcp 10.133.0.169:8000: connect: connection refused"
Oct 31 13:56:15.786349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.782353    4660 generic.go:356] "Generic (PLEG): container finished" podID="7c10f8e2-7ba7-4191-bb14-6a09b346f3a8" containerID="e4017acfc3d16e26a78ff0c47cc41922aeb5c59c0a7dfb499f6620d031ebbe4f" exitCode=0
Oct 31 13:56:15.786349 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.783437    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m" event={"ID":"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8","Type":"ContainerDied","Data":"e4017acfc3d16e26a78ff0c47cc41922aeb5c59c0a7dfb499f6620d031ebbe4f"}
Oct 31 13:56:15.878357 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.872988    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:56:15.958335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.958190    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-tls-certs\") pod \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") "
Oct 31 13:56:15.958677 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.958419    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-kube-api-access-wtxpv\" (UniqueName: \"kubernetes.io/projected/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-kube-api-access-wtxpv\") pod \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") "
Oct 31 13:56:15.958677 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.958608    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-dshm\") pod \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") "
Oct 31 13:56:15.958984 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.958925    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-model-cache\") pod \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") "
Oct 31 13:56:15.959125 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.959094    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-home\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-home\") pod \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") "
Oct 31 13:56:15.959233 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.959214    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-tmp-dir\") pod \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\" (UID: \"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8\") "
Oct 31 13:56:15.959829 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.959795    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-home" pod "7c10f8e2-7ba7-4191-bb14-6a09b346f3a8" (UID: "7c10f8e2-7ba7-4191-bb14-6a09b346f3a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:15.959952 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.959828    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-model-cache" pod "7c10f8e2-7ba7-4191-bb14-6a09b346f3a8" (UID: "7c10f8e2-7ba7-4191-bb14-6a09b346f3a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:15.960025 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.959965    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:15.960025 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.959986    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:15.960187 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.960160    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-tmp-dir" pod "7c10f8e2-7ba7-4191-bb14-6a09b346f3a8" (UID: "7c10f8e2-7ba7-4191-bb14-6a09b346f3a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:15.962136 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.962108    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-tls-certs" pod "7c10f8e2-7ba7-4191-bb14-6a09b346f3a8" (UID: "7c10f8e2-7ba7-4191-bb14-6a09b346f3a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:56:15.962224 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.962196    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-dshm" pod "7c10f8e2-7ba7-4191-bb14-6a09b346f3a8" (UID: "7c10f8e2-7ba7-4191-bb14-6a09b346f3a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:15.962271 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:15.962229    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-kube-api-access-wtxpv" pod "7c10f8e2-7ba7-4191-bb14-6a09b346f3a8" (UID: "7c10f8e2-7ba7-4191-bb14-6a09b346f3a8"). InnerVolumeSpecName "kube-api-access-wtxpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:56:16.061205 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:16.061128    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:16.061205 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:16.061157    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:16.061205 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:16.061168    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtxpv\" (UniqueName: \"kubernetes.io/projected/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-kube-api-access-wtxpv\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:16.061205 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:16.061177    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:16.600485 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:16.600447    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"]
Oct 31 13:56:16.600959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:16.600702    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm" podUID="faa53f5f-46bc-443c-a9ad-4382eec12895" containerName="istio-proxy" containerID="cri-o://7046d92a75745ce1b5cf75e391f2765608877098d1d12f3907fead0e00c6d741" gracePeriod=30
Oct 31 13:56:16.821367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:16.818343    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"
Oct 31 13:56:16.821367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:16.818968    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m" event={"ID":"7c10f8e2-7ba7-4191-bb14-6a09b346f3a8","Type":"ContainerDied","Data":"ec1d3593c02d48618bd69dd222bcebfe4a2526448f5b8e8b12df9b8acbc914ca"}
Oct 31 13:56:16.821367 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:16.819012    4660 scope.go:122] "RemoveContainer" containerID="e4017acfc3d16e26a78ff0c47cc41922aeb5c59c0a7dfb499f6620d031ebbe4f"
Oct 31 13:56:16.877305 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:16.877171    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"]
Oct 31 13:56:16.932288 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:16.929017    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-97927b/sub-test-model-97927b-kserve-76569c7ff7-twn9m"]
Oct 31 13:56:16.997957 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:16.971348    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"]
Oct 31 13:56:16.997957 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:16.972405    4660 cpu_manager.go:414] "containerMap: removing container" podUID="7c10f8e2-7ba7-4191-bb14-6a09b346f3a8" containerName="main"
Oct 31 13:56:16.997957 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:16.972428    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c10f8e2-7ba7-4191-bb14-6a09b346f3a8" containerName="main"
Oct 31 13:56:16.997957 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:16.972648    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c10f8e2-7ba7-4191-bb14-6a09b346f3a8" containerName="main"
Oct 31 13:56:17.006167 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.002199    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"]
Oct 31 13:56:17.006358 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.006344    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.017570 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.017540    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-14ba5a-openshift-default-dockercfg-qj6n9\""
Oct 31 13:56:17.113424 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.105374    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-envoy\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.113424 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.105437    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-token\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.113424 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.105494    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-workload-certs\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.113424 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.105583    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwn95\" (UniqueName: \"kubernetes.io/projected/5db28b27-5dcf-4468-8ebc-951e8f52613f-kube-api-access-cwn95\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.113424 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.105642    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-workload-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.113424 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.105669    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-podinfo\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.113424 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.105920    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db28b27-5dcf-4468-8ebc-951e8f52613f-istiod-ca-cert\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.113424 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.105994    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-data\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.113424 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.106026    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-credential-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.208365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.208328    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db28b27-5dcf-4468-8ebc-951e8f52613f-istiod-ca-cert\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.208581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.208391    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-data\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.208581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.208427    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-credential-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.208581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.208520    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-envoy\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.208581 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.208551    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-token\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.208847 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.208597    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-workload-certs\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.208847 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.208676    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwn95\" (UniqueName: \"kubernetes.io/projected/5db28b27-5dcf-4468-8ebc-951e8f52613f-kube-api-access-cwn95\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.208847 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.208732    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-workload-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.208847 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.208757    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-podinfo\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.209366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.209329    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db28b27-5dcf-4468-8ebc-951e8f52613f-istiod-ca-cert\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.209671 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.209647    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-data\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.209906 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.209887    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-credential-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.210090 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.209949    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-workload-certs\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.210206 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.210083    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-workload-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.215522 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.215495    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-envoy\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.217785 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.217759    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-podinfo\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.227662 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.227634    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-token\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.228480 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.228458    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwn95\" (UniqueName: \"kubernetes.io/projected/5db28b27-5dcf-4468-8ebc-951e8f52613f-kube-api-access-cwn95\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.341817 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.341780    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:17.648847 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.648817    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"]
Oct 31 13:56:17.649687 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:56:17.649655    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db28b27_5dcf_4468_8ebc_951e8f52613f.slice/crio-4c8c0364694170285c671c64996bbaf4826114a4002bc95e4f16f2cc4905941c WatchSource:0}: Error finding container 4c8c0364694170285c671c64996bbaf4826114a4002bc95e4f16f2cc4905941c: Status 404 returned error can't find the container with id 4c8c0364694170285c671c64996bbaf4826114a4002bc95e4f16f2cc4905941c
Oct 31 13:56:17.653487 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.653446    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:56:17.653591 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.653532    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:56:17.653646 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.653630    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:56:17.831084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.830080    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f" event={"ID":"5db28b27-5dcf-4468-8ebc-951e8f52613f","Type":"ContainerStarted","Data":"5b5e52e31c0d6f436e75c900fb2e6e8c8d1b5898b895b391d96383255608870a"}
Oct 31 13:56:17.831084 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.830153    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f" event={"ID":"5db28b27-5dcf-4468-8ebc-951e8f52613f","Type":"ContainerStarted","Data":"4c8c0364694170285c671c64996bbaf4826114a4002bc95e4f16f2cc4905941c"}
Oct 31 13:56:17.859004 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:17.858946    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f" podStartSLOduration=1.8589317429999999 podStartE2EDuration="1.858931743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:56:17.854624851 +0000 UTC m=+3177.473284006" watchObservedRunningTime="2026-10-31 13:56:17.858931743 +0000 UTC m=+3177.477590906"
Oct 31 13:56:18.342390 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:18.342342    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:18.344636 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:18.344603    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 31 13:56:18.344778 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:18.344678    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f" podUID="5db28b27-5dcf-4468-8ebc-951e8f52613f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 31 13:56:18.547583 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:18.547544    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c10f8e2-7ba7-4191-bb14-6a09b346f3a8" path="/var/lib/kubelet/pods/7c10f8e2-7ba7-4191-bb14-6a09b346f3a8/volumes"
Oct 31 13:56:19.241319 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.240354    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"]
Oct 31 13:56:19.248339 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.247556    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.255011 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.254984    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-97927b-openshift-default-dockercfg-p88tk\""
Oct 31 13:56:19.264035 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.262612    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"]
Oct 31 13:56:19.347685 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.347649    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 31 13:56:19.347972 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.347946    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f" podUID="5db28b27-5dcf-4468-8ebc-951e8f52613f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 31 13:56:19.373534 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.373030    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a94c907a-8bdd-4b73-86ee-3c132831df1f-istiod-ca-cert\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.373828 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.373807    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-credential-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.373947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.373933    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-data\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.388524 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.388484    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-podinfo\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.408352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.388766    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkhb7\" (UniqueName: \"kubernetes.io/projected/a94c907a-8bdd-4b73-86ee-3c132831df1f-kube-api-access-fkhb7\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.408719 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.408687    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-workload-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.408919 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.408901    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-envoy\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.409081 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.409065    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-workload-certs\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.409284 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.409271    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-token\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.529206 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.529106    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a94c907a-8bdd-4b73-86ee-3c132831df1f-istiod-ca-cert\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.529206 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.529187    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-credential-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.529456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.529217    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-data\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.529456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.529257    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-podinfo\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.529456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.529305    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkhb7\" (UniqueName: \"kubernetes.io/projected/a94c907a-8bdd-4b73-86ee-3c132831df1f-kube-api-access-fkhb7\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.529456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.529371    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-workload-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.529456 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.529410    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-envoy\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.529714 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.529470    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-workload-certs\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.529714 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.529554    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-token\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.544237 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.530831    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a94c907a-8bdd-4b73-86ee-3c132831df1f-istiod-ca-cert\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.544237 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.531131    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-credential-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.544237 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.542567    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-data\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.544237 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.544161    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-workload-certs\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.545023 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.544999    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-workload-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.546611 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.546580    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-token\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.547542 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.547507    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-podinfo\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.547785 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.547761    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-envoy\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.557120 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.557094    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkhb7\" (UniqueName: \"kubernetes.io/projected/a94c907a-8bdd-4b73-86ee-3c132831df1f-kube-api-access-fkhb7\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.570000 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.569979    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:19.917318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.915381    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"]
Oct 31 13:56:19.937447 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:56:19.937406    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94c907a_8bdd_4b73_86ee_3c132831df1f.slice/crio-0a7ef347daec1e156d100e035aa71238523687802cc3a054ccb91c85d8744226 WatchSource:0}: Error finding container 0a7ef347daec1e156d100e035aa71238523687802cc3a054ccb91c85d8744226: Status 404 returned error can't find the container with id 0a7ef347daec1e156d100e035aa71238523687802cc3a054ccb91c85d8744226
Oct 31 13:56:19.962700 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.962621    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:56:19.962955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.962925    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:56:19.963526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:19.963097    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 13:56:20.378318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:20.371464    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 31 13:56:20.378318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:20.371529    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f" podUID="5db28b27-5dcf-4468-8ebc-951e8f52613f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 31 13:56:20.782491 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:56:20.782456    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:56:20.868032 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:20.867982    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv" event={"ID":"a94c907a-8bdd-4b73-86ee-3c132831df1f","Type":"ContainerStarted","Data":"225651aaf07d0fb1509bf60dadc15531f7ce891a134aa4486a90157200af6e40"}
Oct 31 13:56:20.868198 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:20.868051    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv" event={"ID":"a94c907a-8bdd-4b73-86ee-3c132831df1f","Type":"ContainerStarted","Data":"0a7ef347daec1e156d100e035aa71238523687802cc3a054ccb91c85d8744226"}
Oct 31 13:56:20.898317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:20.897268    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv" podStartSLOduration=1.897249168 podStartE2EDuration="1.897249168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:56:20.893908826 +0000 UTC m=+3180.512567993" watchObservedRunningTime="2026-10-31 13:56:20.897249168 +0000 UTC m=+3180.515908334"
Oct 31 13:56:21.344498 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:21.343680    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 31 13:56:21.344498 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:21.343740    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f" podUID="5db28b27-5dcf-4468-8ebc-951e8f52613f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 31 13:56:21.466183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:21.466139    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:56:21.466715 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:21.466206    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm" podUID="faa53f5f-46bc-443c-a9ad-4382eec12895" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:56:21.570214 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:21.570180    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:21.571782 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:21.571755    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 13:56:21.571918 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:21.571802    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv" podUID="a94c907a-8bdd-4b73-86ee-3c132831df1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 13:56:21.997006 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:21.996878    4660 generic.go:356] "Generic (PLEG): container finished" podID="faa53f5f-46bc-443c-a9ad-4382eec12895" containerID="7046d92a75745ce1b5cf75e391f2765608877098d1d12f3907fead0e00c6d741" exitCode=0
Oct 31 13:56:21.998501 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:21.998421    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm" event={"ID":"faa53f5f-46bc-443c-a9ad-4382eec12895","Type":"ContainerDied","Data":"7046d92a75745ce1b5cf75e391f2765608877098d1d12f3907fead0e00c6d741"}
Oct 31 13:56:22.367378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.366862    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 31 13:56:22.367378 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.366965    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f" podUID="5db28b27-5dcf-4468-8ebc-951e8f52613f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 31 13:56:22.576689 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.576222    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 13:56:22.576689 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.576285    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv" podUID="a94c907a-8bdd-4b73-86ee-3c132831df1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 13:56:22.689282 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.689252    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:56:22.693645 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.693619    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-data\") pod \"faa53f5f-46bc-443c-a9ad-4382eec12895\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") "
Oct 31 13:56:22.693776 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.693676    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-credential-socket\") pod \"faa53f5f-46bc-443c-a9ad-4382eec12895\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") "
Oct 31 13:56:22.693776 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.693748    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-workload-certs\") pod \"faa53f5f-46bc-443c-a9ad-4382eec12895\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") "
Oct 31 13:56:22.693852 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.693807    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-envoy\") pod \"faa53f5f-46bc-443c-a9ad-4382eec12895\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") "
Oct 31 13:56:22.693888 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.693874    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-token\" (UniqueName: \"kubernetes.io/projected/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-token\") pod \"faa53f5f-46bc-443c-a9ad-4382eec12895\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") "
Oct 31 13:56:22.693927 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.693900    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-data" pod "faa53f5f-46bc-443c-a9ad-4382eec12895" (UID: "faa53f5f-46bc-443c-a9ad-4382eec12895"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:22.693927 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.693910    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa53f5f-46bc-443c-a9ad-4382eec12895-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa53f5f-46bc-443c-a9ad-4382eec12895-istiod-ca-cert\") pod \"faa53f5f-46bc-443c-a9ad-4382eec12895\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") "
Oct 31 13:56:22.694066 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.694044    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-workload-socket\") pod \"faa53f5f-46bc-443c-a9ad-4382eec12895\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") "
Oct 31 13:56:22.694141 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.694127    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa53f5f-46bc-443c-a9ad-4382eec12895-kube-api-access-hlw66\" (UniqueName: \"kubernetes.io/projected/faa53f5f-46bc-443c-a9ad-4382eec12895-kube-api-access-hlw66\") pod \"faa53f5f-46bc-443c-a9ad-4382eec12895\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") "
Oct 31 13:56:22.694326 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.694280    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-podinfo\") pod \"faa53f5f-46bc-443c-a9ad-4382eec12895\" (UID: \"faa53f5f-46bc-443c-a9ad-4382eec12895\") "
Oct 31 13:56:22.694654 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.694528    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-workload-socket" pod "faa53f5f-46bc-443c-a9ad-4382eec12895" (UID: "faa53f5f-46bc-443c-a9ad-4382eec12895"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:22.694654 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.694544    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-credential-socket" pod "faa53f5f-46bc-443c-a9ad-4382eec12895" (UID: "faa53f5f-46bc-443c-a9ad-4382eec12895"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:22.694804 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.694785    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-workload-certs" pod "faa53f5f-46bc-443c-a9ad-4382eec12895" (UID: "faa53f5f-46bc-443c-a9ad-4382eec12895"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:22.694958 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.694937    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa53f5f-46bc-443c-a9ad-4382eec12895-istiod-ca-cert" pod "faa53f5f-46bc-443c-a9ad-4382eec12895" (UID: "faa53f5f-46bc-443c-a9ad-4382eec12895"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:56:22.695160 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.695143    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.695278 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.695262    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.695412 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.695397    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.695505 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.695493    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa53f5f-46bc-443c-a9ad-4382eec12895-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.695597 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.695583    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.696830 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.696802    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa53f5f-46bc-443c-a9ad-4382eec12895-kube-api-access-hlw66" pod "faa53f5f-46bc-443c-a9ad-4382eec12895" (UID: "faa53f5f-46bc-443c-a9ad-4382eec12895"). InnerVolumeSpecName "kube-api-access-hlw66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:56:22.696933 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.696918    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-podinfo" pod "faa53f5f-46bc-443c-a9ad-4382eec12895" (UID: "faa53f5f-46bc-443c-a9ad-4382eec12895"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:56:22.697113 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.697095    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-envoy" pod "faa53f5f-46bc-443c-a9ad-4382eec12895" (UID: "faa53f5f-46bc-443c-a9ad-4382eec12895"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:22.697270 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.697250    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-token" pod "faa53f5f-46bc-443c-a9ad-4382eec12895" (UID: "faa53f5f-46bc-443c-a9ad-4382eec12895"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:56:22.797322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.797251    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.797322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.797284    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.797322 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.797315    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlw66\" (UniqueName: \"kubernetes.io/projected/faa53f5f-46bc-443c-a9ad-4382eec12895-kube-api-access-hlw66\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.797584 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:22.797331    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa53f5f-46bc-443c-a9ad-4382eec12895-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:23.098906 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:23.098817    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"
Oct 31 13:56:23.099248 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:23.098809    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm" event={"ID":"faa53f5f-46bc-443c-a9ad-4382eec12895","Type":"ContainerDied","Data":"68be1037116bd8da9e9f67b13a8b4131419a5093669ca4d8da76e433d192dbb3"}
Oct 31 13:56:23.099248 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:23.099000    4660 scope.go:122] "RemoveContainer" containerID="7046d92a75745ce1b5cf75e391f2765608877098d1d12f3907fead0e00c6d741"
Oct 31 13:56:23.200679 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:23.200645    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"]
Oct 31 13:56:23.261320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:23.261212    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-v6bjm"]
Oct 31 13:56:23.354370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:23.349695    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 31 13:56:23.354370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:23.349765    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f" podUID="5db28b27-5dcf-4468-8ebc-951e8f52613f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 31 13:56:23.570836 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:23.570794    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 13:56:23.571020 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:23.570849    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv" podUID="a94c907a-8bdd-4b73-86ee-3c132831df1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 13:56:24.352313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:24.351884    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 31 13:56:24.352313 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:24.351966    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f" podUID="5db28b27-5dcf-4468-8ebc-951e8f52613f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 31 13:56:24.554318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:24.553529    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa53f5f-46bc-443c-a9ad-4382eec12895" path="/var/lib/kubelet/pods/faa53f5f-46bc-443c-a9ad-4382eec12895/volumes"
Oct 31 13:56:24.571450 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:24.571101    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 13:56:24.571450 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:24.571169    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv" podUID="a94c907a-8bdd-4b73-86ee-3c132831df1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 13:56:25.315318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.302754    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss" podUID="d6bad0c4-bf7f-4f21-a957-8014c12c3aba" containerName="main" containerID="cri-o://ece27ab1b7e0e714e5594adc01974413cd3ce9eedd28cd3299ad880f76f16db0" gracePeriod=45
Oct 31 13:56:25.358586 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.358470    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:25.359228 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.359210    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:25.360035 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.360017    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 13:56:25.570571 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.570532    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 13:56:25.570737 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.570600    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv" podUID="a94c907a-8bdd-4b73-86ee-3c132831df1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 13:56:25.773639 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.773611    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:56:25.882006 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.881963    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-kube-api-access-sd4c2\" (UniqueName: \"kubernetes.io/projected/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-kube-api-access-sd4c2\") pod \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") "
Oct 31 13:56:25.882183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.882056    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-model-cache\") pod \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") "
Oct 31 13:56:25.882183 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.882144    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-tmp-dir\") pod \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") "
Oct 31 13:56:25.882273 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.882202    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-dshm\") pod \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") "
Oct 31 13:56:25.882330 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.882273    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-tls-certs\") pod \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") "
Oct 31 13:56:25.882489 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.882471    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-home\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-home\") pod \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\" (UID: \"d6bad0c4-bf7f-4f21-a957-8014c12c3aba\") "
Oct 31 13:56:25.884133 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.884096    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-model-cache" pod "d6bad0c4-bf7f-4f21-a957-8014c12c3aba" (UID: "d6bad0c4-bf7f-4f21-a957-8014c12c3aba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:25.884686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.884659    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-home" pod "d6bad0c4-bf7f-4f21-a957-8014c12c3aba" (UID: "d6bad0c4-bf7f-4f21-a957-8014c12c3aba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:25.884906 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.884886    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-tmp-dir" pod "d6bad0c4-bf7f-4f21-a957-8014c12c3aba" (UID: "d6bad0c4-bf7f-4f21-a957-8014c12c3aba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:25.886327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.886267    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-kube-api-access-sd4c2" pod "d6bad0c4-bf7f-4f21-a957-8014c12c3aba" (UID: "d6bad0c4-bf7f-4f21-a957-8014c12c3aba"). InnerVolumeSpecName "kube-api-access-sd4c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:56:25.887767 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.887742    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-tls-certs" pod "d6bad0c4-bf7f-4f21-a957-8014c12c3aba" (UID: "d6bad0c4-bf7f-4f21-a957-8014c12c3aba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:56:25.890167 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.890144    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-dshm" pod "d6bad0c4-bf7f-4f21-a957-8014c12c3aba" (UID: "d6bad0c4-bf7f-4f21-a957-8014c12c3aba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:25.985600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.985564    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:25.985600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.985597    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:25.985820 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.985630    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:25.985820 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.985644    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:25.985820 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.985657    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd4c2\" (UniqueName: \"kubernetes.io/projected/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-kube-api-access-sd4c2\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:25.985820 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:25.985670    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bad0c4-bf7f-4f21-a957-8014c12c3aba-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:26.372993 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:26.372959    4660 generic.go:356] "Generic (PLEG): container finished" podID="d6bad0c4-bf7f-4f21-a957-8014c12c3aba" containerID="ece27ab1b7e0e714e5594adc01974413cd3ce9eedd28cd3299ad880f76f16db0" exitCode=0
Oct 31 13:56:26.373501 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:26.373015    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"
Oct 31 13:56:26.373651 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:26.373622    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss" event={"ID":"d6bad0c4-bf7f-4f21-a957-8014c12c3aba","Type":"ContainerDied","Data":"ece27ab1b7e0e714e5594adc01974413cd3ce9eedd28cd3299ad880f76f16db0"}
Oct 31 13:56:26.373813 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:26.373778    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss" event={"ID":"d6bad0c4-bf7f-4f21-a957-8014c12c3aba","Type":"ContainerDied","Data":"765873c725a540ddde415a8b0d2ab3566f7962254a1152ba7608d4b6bbffc349"}
Oct 31 13:56:26.373947 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:26.373815    4660 scope.go:122] "RemoveContainer" containerID="ece27ab1b7e0e714e5594adc01974413cd3ce9eedd28cd3299ad880f76f16db0"
Oct 31 13:56:26.389248 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:26.389227    4660 scope.go:122] "RemoveContainer" containerID="ece27ab1b7e0e714e5594adc01974413cd3ce9eedd28cd3299ad880f76f16db0"
Oct 31 13:56:26.389581 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:56:26.389558    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece27ab1b7e0e714e5594adc01974413cd3ce9eedd28cd3299ad880f76f16db0\": container with ID starting with ece27ab1b7e0e714e5594adc01974413cd3ce9eedd28cd3299ad880f76f16db0 not found: ID does not exist" containerID="ece27ab1b7e0e714e5594adc01974413cd3ce9eedd28cd3299ad880f76f16db0"
Oct 31 13:56:26.389667 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:26.389589    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece27ab1b7e0e714e5594adc01974413cd3ce9eedd28cd3299ad880f76f16db0"} err="failed to get container status \"ece27ab1b7e0e714e5594adc01974413cd3ce9eedd28cd3299ad880f76f16db0\": rpc error: code = NotFound desc = could not find container \"ece27ab1b7e0e714e5594adc01974413cd3ce9eedd28cd3299ad880f76f16db0\": container with ID starting with ece27ab1b7e0e714e5594adc01974413cd3ce9eedd28cd3299ad880f76f16db0 not found: ID does not exist"
Oct 31 13:56:26.417316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:26.413343    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"]
Oct 31 13:56:26.426984 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:26.426947    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-599291/sub-test-model-599291-kserve-7577667d79-z6vss"]
Oct 31 13:56:26.544373 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:26.544329    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6bad0c4-bf7f-4f21-a957-8014c12c3aba" path="/var/lib/kubelet/pods/d6bad0c4-bf7f-4f21-a957-8014c12c3aba/volumes"
Oct 31 13:56:26.571365 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:26.571333    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 13:56:26.571578 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:26.571387    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv" podUID="a94c907a-8bdd-4b73-86ee-3c132831df1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 13:56:27.573330 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:27.573275    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 13:56:27.579755 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:27.576952    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv" podUID="a94c907a-8bdd-4b73-86ee-3c132831df1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 13:56:28.576762 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:28.576727    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:28.577506 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:28.577483    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:28.578156 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:56:28.578137    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 13:56:50.738215 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:56:50.738173    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:57:20.735197 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:57:20.735154    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:57:28.873118 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:28.873070    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"]
Oct 31 13:57:30.994438 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:30.994392    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"]
Oct 31 13:57:31.007234 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:31.007206    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"]
Oct 31 13:57:33.584512 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:33.584453    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"]
Oct 31 13:57:33.585533 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:33.585495    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg" podUID="cec915fa-dbf8-43f5-bb74-db141931047c" containerName="maas-api" containerID="cri-o://63d743d66f41a33257ceacb41a9195e20cba6da4d2cd34f1521dd626504e396f" gracePeriod=30
Oct 31 13:57:34.359340 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.359312    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:57:34.525020 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.524978    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cec915fa-dbf8-43f5-bb74-db141931047c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec915fa-dbf8-43f5-bb74-db141931047c-metrics-certs\") pod \"cec915fa-dbf8-43f5-bb74-db141931047c\" (UID: \"cec915fa-dbf8-43f5-bb74-db141931047c\") "
Oct 31 13:57:34.525232 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.525130    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cec915fa-dbf8-43f5-bb74-db141931047c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cec915fa-dbf8-43f5-bb74-db141931047c-maas-api-tls\") pod \"cec915fa-dbf8-43f5-bb74-db141931047c\" (UID: \"cec915fa-dbf8-43f5-bb74-db141931047c\") "
Oct 31 13:57:34.525232 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.525197    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cec915fa-dbf8-43f5-bb74-db141931047c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cec915fa-dbf8-43f5-bb74-db141931047c-openshift-service-ca\") pod \"cec915fa-dbf8-43f5-bb74-db141931047c\" (UID: \"cec915fa-dbf8-43f5-bb74-db141931047c\") "
Oct 31 13:57:34.525511 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.525417    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec915fa-dbf8-43f5-bb74-db141931047c-kube-api-access-v9q2b\" (UniqueName: \"kubernetes.io/projected/cec915fa-dbf8-43f5-bb74-db141931047c-kube-api-access-v9q2b\") pod \"cec915fa-dbf8-43f5-bb74-db141931047c\" (UID: \"cec915fa-dbf8-43f5-bb74-db141931047c\") "
Oct 31 13:57:34.528028 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.527995    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec915fa-dbf8-43f5-bb74-db141931047c-openshift-service-ca" pod "cec915fa-dbf8-43f5-bb74-db141931047c" (UID: "cec915fa-dbf8-43f5-bb74-db141931047c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:57:34.532523 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.532473    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cec915fa-dbf8-43f5-bb74-db141931047c-metrics-certs" pod "cec915fa-dbf8-43f5-bb74-db141931047c" (UID: "cec915fa-dbf8-43f5-bb74-db141931047c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:57:34.538366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.538338    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cec915fa-dbf8-43f5-bb74-db141931047c-maas-api-tls" pod "cec915fa-dbf8-43f5-bb74-db141931047c" (UID: "cec915fa-dbf8-43f5-bb74-db141931047c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:57:34.538492 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.538469    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec915fa-dbf8-43f5-bb74-db141931047c-kube-api-access-v9q2b" pod "cec915fa-dbf8-43f5-bb74-db141931047c" (UID: "cec915fa-dbf8-43f5-bb74-db141931047c"). InnerVolumeSpecName "kube-api-access-v9q2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:57:34.630613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.630582    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9q2b\" (UniqueName: \"kubernetes.io/projected/cec915fa-dbf8-43f5-bb74-db141931047c-kube-api-access-v9q2b\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:34.631057 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.631038    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec915fa-dbf8-43f5-bb74-db141931047c-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:34.631180 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.631165    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cec915fa-dbf8-43f5-bb74-db141931047c-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:34.631335 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.631281    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cec915fa-dbf8-43f5-bb74-db141931047c-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:34.729323 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.729253    4660 generic.go:356] "Generic (PLEG): container finished" podID="cec915fa-dbf8-43f5-bb74-db141931047c" containerID="63d743d66f41a33257ceacb41a9195e20cba6da4d2cd34f1521dd626504e396f" exitCode=0
Oct 31 13:57:34.729518 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.729361    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"
Oct 31 13:57:34.729518 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.729355    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg" event={"ID":"cec915fa-dbf8-43f5-bb74-db141931047c","Type":"ContainerDied","Data":"63d743d66f41a33257ceacb41a9195e20cba6da4d2cd34f1521dd626504e396f"}
Oct 31 13:57:34.729518 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.729484    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg" event={"ID":"cec915fa-dbf8-43f5-bb74-db141931047c","Type":"ContainerDied","Data":"c06ecba01373bae0f04196c965eaca1180561e02a62484700cb0798020925df9"}
Oct 31 13:57:34.729518 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.729510    4660 scope.go:122] "RemoveContainer" containerID="63d743d66f41a33257ceacb41a9195e20cba6da4d2cd34f1521dd626504e396f"
Oct 31 13:57:34.741541 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.741516    4660 scope.go:122] "RemoveContainer" containerID="63d743d66f41a33257ceacb41a9195e20cba6da4d2cd34f1521dd626504e396f"
Oct 31 13:57:34.741864 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:57:34.741831    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d743d66f41a33257ceacb41a9195e20cba6da4d2cd34f1521dd626504e396f\": container with ID starting with 63d743d66f41a33257ceacb41a9195e20cba6da4d2cd34f1521dd626504e396f not found: ID does not exist" containerID="63d743d66f41a33257ceacb41a9195e20cba6da4d2cd34f1521dd626504e396f"
Oct 31 13:57:34.741964 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.741875    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d743d66f41a33257ceacb41a9195e20cba6da4d2cd34f1521dd626504e396f"} err="failed to get container status \"63d743d66f41a33257ceacb41a9195e20cba6da4d2cd34f1521dd626504e396f\": rpc error: code = NotFound desc = could not find container \"63d743d66f41a33257ceacb41a9195e20cba6da4d2cd34f1521dd626504e396f\": container with ID starting with 63d743d66f41a33257ceacb41a9195e20cba6da4d2cd34f1521dd626504e396f not found: ID does not exist"
Oct 31 13:57:34.754195 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.754164    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"]
Oct 31 13:57:34.758640 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:34.758616    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dafbce-57bd5fdccb-bbzhg"]
Oct 31 13:57:35.489314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:35.481159    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"]
Oct 31 13:57:36.542315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:36.541078    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cec915fa-dbf8-43f5-bb74-db141931047c" path="/var/lib/kubelet/pods/cec915fa-dbf8-43f5-bb74-db141931047c/volumes"
Oct 31 13:57:44.019350 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.019264    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b" podUID="cdb2b8e0-12fa-4b14-ac14-75e2aade9b80" containerName="main" containerID="cri-o://5d11a1e1b8edcc4cb81b460d3d95ff8c56aaab5800f558bef177f2caa0b5d7b9" gracePeriod=45
Oct 31 13:57:44.355037 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.355005    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:57:44.452275 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.452239    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-tls-certs\") pod \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") "
Oct 31 13:57:44.452494 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.452428    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-tmp-dir\") pod \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") "
Oct 31 13:57:44.452494 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.452482    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-kube-api-access-n97kt\" (UniqueName: \"kubernetes.io/projected/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-kube-api-access-n97kt\") pod \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") "
Oct 31 13:57:44.452632 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.452519    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-home\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-home\") pod \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") "
Oct 31 13:57:44.452632 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.452548    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-model-cache\") pod \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") "
Oct 31 13:57:44.452632 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.452577    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-dshm\") pod \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\" (UID: \"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80\") "
Oct 31 13:57:44.452812 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.452745    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-tmp-dir" pod "cdb2b8e0-12fa-4b14-ac14-75e2aade9b80" (UID: "cdb2b8e0-12fa-4b14-ac14-75e2aade9b80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:44.453208 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.453117    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-home" pod "cdb2b8e0-12fa-4b14-ac14-75e2aade9b80" (UID: "cdb2b8e0-12fa-4b14-ac14-75e2aade9b80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:44.453208 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.453172    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-model-cache" pod "cdb2b8e0-12fa-4b14-ac14-75e2aade9b80" (UID: "cdb2b8e0-12fa-4b14-ac14-75e2aade9b80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:44.453437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.453354    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:44.453437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.453374    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:44.453437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.453387    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:44.455114 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.454987    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-tls-certs" pod "cdb2b8e0-12fa-4b14-ac14-75e2aade9b80" (UID: "cdb2b8e0-12fa-4b14-ac14-75e2aade9b80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:57:44.455240 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.455125    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-dshm" pod "cdb2b8e0-12fa-4b14-ac14-75e2aade9b80" (UID: "cdb2b8e0-12fa-4b14-ac14-75e2aade9b80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:44.455360 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.455334    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-kube-api-access-n97kt" pod "cdb2b8e0-12fa-4b14-ac14-75e2aade9b80" (UID: "cdb2b8e0-12fa-4b14-ac14-75e2aade9b80"). InnerVolumeSpecName "kube-api-access-n97kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:57:44.555370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.555341    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n97kt\" (UniqueName: \"kubernetes.io/projected/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-kube-api-access-n97kt\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:44.555370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.555372    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:44.555547 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.555386    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:44.829859 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.829812    4660 generic.go:356] "Generic (PLEG): container finished" podID="cdb2b8e0-12fa-4b14-ac14-75e2aade9b80" containerID="5d11a1e1b8edcc4cb81b460d3d95ff8c56aaab5800f558bef177f2caa0b5d7b9" exitCode=0
Oct 31 13:57:44.830042 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.829870    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"
Oct 31 13:57:44.830042 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.829870    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b" event={"ID":"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80","Type":"ContainerDied","Data":"5d11a1e1b8edcc4cb81b460d3d95ff8c56aaab5800f558bef177f2caa0b5d7b9"}
Oct 31 13:57:44.830042 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.829921    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b" event={"ID":"cdb2b8e0-12fa-4b14-ac14-75e2aade9b80","Type":"ContainerDied","Data":"edab8b1a162dd6fd853dc4b44a9f72b9a2cb39dd958c02ac4303539de9660943"}
Oct 31 13:57:44.830042 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.829943    4660 scope.go:122] "RemoveContainer" containerID="5d11a1e1b8edcc4cb81b460d3d95ff8c56aaab5800f558bef177f2caa0b5d7b9"
Oct 31 13:57:44.847715 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.847689    4660 scope.go:122] "RemoveContainer" containerID="5d11a1e1b8edcc4cb81b460d3d95ff8c56aaab5800f558bef177f2caa0b5d7b9"
Oct 31 13:57:44.848040 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:57:44.848013    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d11a1e1b8edcc4cb81b460d3d95ff8c56aaab5800f558bef177f2caa0b5d7b9\": container with ID starting with 5d11a1e1b8edcc4cb81b460d3d95ff8c56aaab5800f558bef177f2caa0b5d7b9 not found: ID does not exist" containerID="5d11a1e1b8edcc4cb81b460d3d95ff8c56aaab5800f558bef177f2caa0b5d7b9"
Oct 31 13:57:44.848188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.848052    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d11a1e1b8edcc4cb81b460d3d95ff8c56aaab5800f558bef177f2caa0b5d7b9"} err="failed to get container status \"5d11a1e1b8edcc4cb81b460d3d95ff8c56aaab5800f558bef177f2caa0b5d7b9\": rpc error: code = NotFound desc = could not find container \"5d11a1e1b8edcc4cb81b460d3d95ff8c56aaab5800f558bef177f2caa0b5d7b9\": container with ID starting with 5d11a1e1b8edcc4cb81b460d3d95ff8c56aaab5800f558bef177f2caa0b5d7b9 not found: ID does not exist"
Oct 31 13:57:44.863398 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.863370    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"]
Oct 31 13:57:44.870867 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:44.870841    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-33c10a/rate-test-model-33c10a-kserve-5dfb68fd4f-pdf8b"]
Oct 31 13:57:46.020161 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.020109    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main" containerID="cri-o://16d276aaa0f3fef64b2b5a8fd76a436423af30df7cb05f31423ee0f092a6acf1" gracePeriod=45
Oct 31 13:57:46.030507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.030470    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" containerID="cri-o://7d67759a957948021aa2a55a26421e06d14745abe1dc081365c24a10ffab849c" gracePeriod=45
Oct 31 13:57:46.147395 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.147284    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": dial tcp 10.133.0.75:8000: connect: connection refused"
Oct 31 13:57:46.397728 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.397697    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:57:46.459803 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.459770    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:57:46.498589 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.498553    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-tmp-dir\") pod \"20570d2d-2c07-487b-a561-59e10e351154\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") "
Oct 31 13:57:46.498840 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.498651    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20570d2d-2c07-487b-a561-59e10e351154-kube-api-access-ddwq5\" (UniqueName: \"kubernetes.io/projected/20570d2d-2c07-487b-a561-59e10e351154-kube-api-access-ddwq5\") pod \"20570d2d-2c07-487b-a561-59e10e351154\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") "
Oct 31 13:57:46.498840 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.498683    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-model-cache\") pod \"a8742e18-cae9-470f-8279-a78c237a4c58\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") "
Oct 31 13:57:46.498840 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.498764    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-dshm\") pod \"20570d2d-2c07-487b-a561-59e10e351154\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") "
Oct 31 13:57:46.498840 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.498803    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-home\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-home\") pod \"20570d2d-2c07-487b-a561-59e10e351154\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") "
Oct 31 13:57:46.498840 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.498835    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-tmp-dir\") pod \"a8742e18-cae9-470f-8279-a78c237a4c58\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") "
Oct 31 13:57:46.499127 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.498852    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-tmp-dir" pod "20570d2d-2c07-487b-a561-59e10e351154" (UID: "20570d2d-2c07-487b-a561-59e10e351154"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:46.499127 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.498886    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20570d2d-2c07-487b-a561-59e10e351154-tls-certs\" (UniqueName: \"kubernetes.io/secret/20570d2d-2c07-487b-a561-59e10e351154-tls-certs\") pod \"20570d2d-2c07-487b-a561-59e10e351154\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") "
Oct 31 13:57:46.499127 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.498963    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-dshm\") pod \"a8742e18-cae9-470f-8279-a78c237a4c58\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") "
Oct 31 13:57:46.499127 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.499001    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8742e18-cae9-470f-8279-a78c237a4c58-kube-api-access-lzrtr\" (UniqueName: \"kubernetes.io/projected/a8742e18-cae9-470f-8279-a78c237a4c58-kube-api-access-lzrtr\") pod \"a8742e18-cae9-470f-8279-a78c237a4c58\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") "
Oct 31 13:57:46.499351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.499229    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8742e18-cae9-470f-8279-a78c237a4c58-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8742e18-cae9-470f-8279-a78c237a4c58-tls-certs\") pod \"a8742e18-cae9-470f-8279-a78c237a4c58\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") "
Oct 31 13:57:46.499351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.499268    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-home\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-home\") pod \"a8742e18-cae9-470f-8279-a78c237a4c58\" (UID: \"a8742e18-cae9-470f-8279-a78c237a4c58\") "
Oct 31 13:57:46.499351 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.499335    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-model-cache\") pod \"20570d2d-2c07-487b-a561-59e10e351154\" (UID: \"20570d2d-2c07-487b-a561-59e10e351154\") "
Oct 31 13:57:46.499507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.499407    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-tmp-dir" pod "a8742e18-cae9-470f-8279-a78c237a4c58" (UID: "a8742e18-cae9-470f-8279-a78c237a4c58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:46.499627 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.499584    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-model-cache" pod "a8742e18-cae9-470f-8279-a78c237a4c58" (UID: "a8742e18-cae9-470f-8279-a78c237a4c58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:46.500600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.499853    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-model-cache" pod "20570d2d-2c07-487b-a561-59e10e351154" (UID: "20570d2d-2c07-487b-a561-59e10e351154"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:46.500600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.499961    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-home" pod "20570d2d-2c07-487b-a561-59e10e351154" (UID: "20570d2d-2c07-487b-a561-59e10e351154"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:46.500600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.500378    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-home" pod "a8742e18-cae9-470f-8279-a78c237a4c58" (UID: "a8742e18-cae9-470f-8279-a78c237a4c58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:46.500600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.500588    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.500885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.500606    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.500885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.500622    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.500885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.500635    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.500885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.500648    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.500885 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.500662    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.504661 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.504634    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-dshm" pod "20570d2d-2c07-487b-a561-59e10e351154" (UID: "20570d2d-2c07-487b-a561-59e10e351154"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:46.507541 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.507524    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20570d2d-2c07-487b-a561-59e10e351154-tls-certs" pod "20570d2d-2c07-487b-a561-59e10e351154" (UID: "20570d2d-2c07-487b-a561-59e10e351154"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:57:46.507846 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.507770    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8742e18-cae9-470f-8279-a78c237a4c58-kube-api-access-lzrtr" pod "a8742e18-cae9-470f-8279-a78c237a4c58" (UID: "a8742e18-cae9-470f-8279-a78c237a4c58"). InnerVolumeSpecName "kube-api-access-lzrtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:57:46.507846 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.507784    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8742e18-cae9-470f-8279-a78c237a4c58-tls-certs" pod "a8742e18-cae9-470f-8279-a78c237a4c58" (UID: "a8742e18-cae9-470f-8279-a78c237a4c58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:57:46.508135 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.508105    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20570d2d-2c07-487b-a561-59e10e351154-kube-api-access-ddwq5" pod "20570d2d-2c07-487b-a561-59e10e351154" (UID: "20570d2d-2c07-487b-a561-59e10e351154"). InnerVolumeSpecName "kube-api-access-ddwq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:57:46.510377 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.509529    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-dshm" pod "a8742e18-cae9-470f-8279-a78c237a4c58" (UID: "a8742e18-cae9-470f-8279-a78c237a4c58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:46.539002 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.538966    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb2b8e0-12fa-4b14-ac14-75e2aade9b80" path="/var/lib/kubelet/pods/cdb2b8e0-12fa-4b14-ac14-75e2aade9b80/volumes"
Oct 31 13:57:46.603370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.603330    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20570d2d-2c07-487b-a561-59e10e351154-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.603370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.603370    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8742e18-cae9-470f-8279-a78c237a4c58-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.603624 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.603386    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzrtr\" (UniqueName: \"kubernetes.io/projected/a8742e18-cae9-470f-8279-a78c237a4c58-kube-api-access-lzrtr\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.603624 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.603400    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8742e18-cae9-470f-8279-a78c237a4c58-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.603624 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.603414    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddwq5\" (UniqueName: \"kubernetes.io/projected/20570d2d-2c07-487b-a561-59e10e351154-kube-api-access-ddwq5\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.603624 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.603427    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20570d2d-2c07-487b-a561-59e10e351154-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.861720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.861048    4660 generic.go:356] "Generic (PLEG): container finished" podID="a8742e18-cae9-470f-8279-a78c237a4c58" containerID="7d67759a957948021aa2a55a26421e06d14745abe1dc081365c24a10ffab849c" exitCode=0
Oct 31 13:57:46.861720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.861232    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" event={"ID":"a8742e18-cae9-470f-8279-a78c237a4c58","Type":"ContainerDied","Data":"7d67759a957948021aa2a55a26421e06d14745abe1dc081365c24a10ffab849c"}
Oct 31 13:57:46.861720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.861267    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk" event={"ID":"a8742e18-cae9-470f-8279-a78c237a4c58","Type":"ContainerDied","Data":"81295f2b57afd187e9a75292a08f7616e12beab01e5778073c621f3e0d2f590c"}
Oct 31 13:57:46.861720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.861286    4660 scope.go:122] "RemoveContainer" containerID="7d67759a957948021aa2a55a26421e06d14745abe1dc081365c24a10ffab849c"
Oct 31 13:57:46.861720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.861455    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"
Oct 31 13:57:46.867959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.867928    4660 generic.go:356] "Generic (PLEG): container finished" podID="20570d2d-2c07-487b-a561-59e10e351154" containerID="16d276aaa0f3fef64b2b5a8fd76a436423af30df7cb05f31423ee0f092a6acf1" exitCode=0
Oct 31 13:57:46.868113 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.868095    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" event={"ID":"20570d2d-2c07-487b-a561-59e10e351154","Type":"ContainerDied","Data":"16d276aaa0f3fef64b2b5a8fd76a436423af30df7cb05f31423ee0f092a6acf1"}
Oct 31 13:57:46.868198 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.868127    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d" event={"ID":"20570d2d-2c07-487b-a561-59e10e351154","Type":"ContainerDied","Data":"f15c6533315f40ebe87d781911a31b1649846cb041eeec0cfb9f21a38314ff84"}
Oct 31 13:57:46.868198 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.868192    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"
Oct 31 13:57:46.881828 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.881799    4660 scope.go:122] "RemoveContainer" containerID="7d67759a957948021aa2a55a26421e06d14745abe1dc081365c24a10ffab849c"
Oct 31 13:57:46.882148 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:57:46.882128    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d67759a957948021aa2a55a26421e06d14745abe1dc081365c24a10ffab849c\": container with ID starting with 7d67759a957948021aa2a55a26421e06d14745abe1dc081365c24a10ffab849c not found: ID does not exist" containerID="7d67759a957948021aa2a55a26421e06d14745abe1dc081365c24a10ffab849c"
Oct 31 13:57:46.882248 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.882159    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d67759a957948021aa2a55a26421e06d14745abe1dc081365c24a10ffab849c"} err="failed to get container status \"7d67759a957948021aa2a55a26421e06d14745abe1dc081365c24a10ffab849c\": rpc error: code = NotFound desc = could not find container \"7d67759a957948021aa2a55a26421e06d14745abe1dc081365c24a10ffab849c\": container with ID starting with 7d67759a957948021aa2a55a26421e06d14745abe1dc081365c24a10ffab849c not found: ID does not exist"
Oct 31 13:57:46.882248 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.882183    4660 scope.go:122] "RemoveContainer" containerID="16d276aaa0f3fef64b2b5a8fd76a436423af30df7cb05f31423ee0f092a6acf1"
Oct 31 13:57:46.884861 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.884835    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"]
Oct 31 13:57:46.891362 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.891336    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/distinct-w2-5286ea-kserve-9bdd7d864-qfrbk"]
Oct 31 13:57:46.894678 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.894656    4660 scope.go:122] "RemoveContainer" containerID="16d276aaa0f3fef64b2b5a8fd76a436423af30df7cb05f31423ee0f092a6acf1"
Oct 31 13:57:46.894958 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:57:46.894937    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d276aaa0f3fef64b2b5a8fd76a436423af30df7cb05f31423ee0f092a6acf1\": container with ID starting with 16d276aaa0f3fef64b2b5a8fd76a436423af30df7cb05f31423ee0f092a6acf1 not found: ID does not exist" containerID="16d276aaa0f3fef64b2b5a8fd76a436423af30df7cb05f31423ee0f092a6acf1"
Oct 31 13:57:46.895020 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.894964    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d276aaa0f3fef64b2b5a8fd76a436423af30df7cb05f31423ee0f092a6acf1"} err="failed to get container status \"16d276aaa0f3fef64b2b5a8fd76a436423af30df7cb05f31423ee0f092a6acf1\": rpc error: code = NotFound desc = could not find container \"16d276aaa0f3fef64b2b5a8fd76a436423af30df7cb05f31423ee0f092a6acf1\": container with ID starting with 16d276aaa0f3fef64b2b5a8fd76a436423af30df7cb05f31423ee0f092a6acf1 not found: ID does not exist"
Oct 31 13:57:46.902840 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.902818    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"]
Oct 31 13:57:46.933361 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:46.914707    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/distinct-2-w2-5286ea-kserve-598b5fb9d7-6vd9d"]
Oct 31 13:57:48.530799 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:48.530758    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20570d2d-2c07-487b-a561-59e10e351154" path="/var/lib/kubelet/pods/20570d2d-2c07-487b-a561-59e10e351154/volumes"
Oct 31 13:57:48.531515 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:48.531492    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" path="/var/lib/kubelet/pods/a8742e18-cae9-470f-8279-a78c237a4c58/volumes"
Oct 31 13:57:50.514830 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.514764    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62" podUID="1b84ce8f-b99a-42bd-b684-ae26a52e8b3f" containerName="main" containerID="cri-o://c2b1c45708beb0798c8cab1e182c12ff7d4e825553389f6a9fe3909b7c9c5d52" gracePeriod=45
Oct 31 13:57:50.770084 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:57:50.769983    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:57:50.865023 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.864989    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:57:50.892386 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.892350    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-dshm\") pod \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") "
Oct 31 13:57:50.892585 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.892453    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-home\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-home\") pod \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") "
Oct 31 13:57:50.892585 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.892543    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-kube-api-access-j25q8\" (UniqueName: \"kubernetes.io/projected/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-kube-api-access-j25q8\") pod \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") "
Oct 31 13:57:50.892720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.892680    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-model-cache\") pod \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") "
Oct 31 13:57:50.892720 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.892714    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-tls-certs\") pod \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") "
Oct 31 13:57:50.892836 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.892820    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-tmp-dir\") pod \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\" (UID: \"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f\") "
Oct 31 13:57:50.893609 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.893579    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-tmp-dir" pod "1b84ce8f-b99a-42bd-b684-ae26a52e8b3f" (UID: "1b84ce8f-b99a-42bd-b684-ae26a52e8b3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:50.894519 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.894478    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-model-cache" pod "1b84ce8f-b99a-42bd-b684-ae26a52e8b3f" (UID: "1b84ce8f-b99a-42bd-b684-ae26a52e8b3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:50.894691 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.894667    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-home" pod "1b84ce8f-b99a-42bd-b684-ae26a52e8b3f" (UID: "1b84ce8f-b99a-42bd-b684-ae26a52e8b3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:50.896092 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.896060    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-dshm" pod "1b84ce8f-b99a-42bd-b684-ae26a52e8b3f" (UID: "1b84ce8f-b99a-42bd-b684-ae26a52e8b3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:50.896468 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.896438    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-kube-api-access-j25q8" pod "1b84ce8f-b99a-42bd-b684-ae26a52e8b3f" (UID: "1b84ce8f-b99a-42bd-b684-ae26a52e8b3f"). InnerVolumeSpecName "kube-api-access-j25q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:57:50.897430 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.897404    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-tls-certs" pod "1b84ce8f-b99a-42bd-b684-ae26a52e8b3f" (UID: "1b84ce8f-b99a-42bd-b684-ae26a52e8b3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:57:50.948324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.944979    4660 generic.go:356] "Generic (PLEG): container finished" podID="1b84ce8f-b99a-42bd-b684-ae26a52e8b3f" containerID="c2b1c45708beb0798c8cab1e182c12ff7d4e825553389f6a9fe3909b7c9c5d52" exitCode=0
Oct 31 13:57:50.948324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.945216    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62" event={"ID":"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f","Type":"ContainerDied","Data":"c2b1c45708beb0798c8cab1e182c12ff7d4e825553389f6a9fe3909b7c9c5d52"}
Oct 31 13:57:50.948324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.945255    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62" event={"ID":"1b84ce8f-b99a-42bd-b684-ae26a52e8b3f","Type":"ContainerDied","Data":"a2126d7c10934a5b968ed72113b9aa6f97b44ca5dd97f5543f40aca0adb64771"}
Oct 31 13:57:50.948324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.945281    4660 scope.go:122] "RemoveContainer" containerID="c2b1c45708beb0798c8cab1e182c12ff7d4e825553389f6a9fe3909b7c9c5d52"
Oct 31 13:57:50.948324 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.945426    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"
Oct 31 13:57:50.965625 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.965586    4660 scope.go:122] "RemoveContainer" containerID="c2b1c45708beb0798c8cab1e182c12ff7d4e825553389f6a9fe3909b7c9c5d52"
Oct 31 13:57:50.966068 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:57:50.966035    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b1c45708beb0798c8cab1e182c12ff7d4e825553389f6a9fe3909b7c9c5d52\": container with ID starting with c2b1c45708beb0798c8cab1e182c12ff7d4e825553389f6a9fe3909b7c9c5d52 not found: ID does not exist" containerID="c2b1c45708beb0798c8cab1e182c12ff7d4e825553389f6a9fe3909b7c9c5d52"
Oct 31 13:57:50.966166 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.966077    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b1c45708beb0798c8cab1e182c12ff7d4e825553389f6a9fe3909b7c9c5d52"} err="failed to get container status \"c2b1c45708beb0798c8cab1e182c12ff7d4e825553389f6a9fe3909b7c9c5d52\": rpc error: code = NotFound desc = could not find container \"c2b1c45708beb0798c8cab1e182c12ff7d4e825553389f6a9fe3909b7c9c5d52\": container with ID starting with c2b1c45708beb0798c8cab1e182c12ff7d4e825553389f6a9fe3909b7c9c5d52 not found: ID does not exist"
Oct 31 13:57:50.980959 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.979978    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"]
Oct 31 13:57:50.985613 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.985586    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-abdcb1/rate-test-model-abdcb1-kserve-65d65d5f97-56h62"]
Oct 31 13:57:50.994643 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.994620    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:50.994643 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.994644    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:50.994770 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.994654    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j25q8\" (UniqueName: \"kubernetes.io/projected/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-kube-api-access-j25q8\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:50.994770 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.994665    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:50.994770 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.994675    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:50.994770 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:50.994685    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:52.539216 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.539175    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b84ce8f-b99a-42bd-b684-ae26a52e8b3f" path="/var/lib/kubelet/pods/1b84ce8f-b99a-42bd-b684-ae26a52e8b3f/volumes"
Oct 31 13:57:52.719935 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.719898    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"]
Oct 31 13:57:52.720843 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.720821    4660 cpu_manager.go:414] "containerMap: removing container" podUID="faa53f5f-46bc-443c-a9ad-4382eec12895" containerName="istio-proxy"
Oct 31 13:57:52.721002 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.720990    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa53f5f-46bc-443c-a9ad-4382eec12895" containerName="istio-proxy"
Oct 31 13:57:52.721112 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.721102    4660 cpu_manager.go:414] "containerMap: removing container" podUID="cdb2b8e0-12fa-4b14-ac14-75e2aade9b80" containerName="main"
Oct 31 13:57:52.721188 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.721177    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb2b8e0-12fa-4b14-ac14-75e2aade9b80" containerName="main"
Oct 31 13:57:52.721307 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.721282    4660 cpu_manager.go:414] "containerMap: removing container" podUID="d6bad0c4-bf7f-4f21-a957-8014c12c3aba" containerName="main"
Oct 31 13:57:52.721399 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.721387    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6bad0c4-bf7f-4f21-a957-8014c12c3aba" containerName="main"
Oct 31 13:57:52.721493 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.721484    4660 cpu_manager.go:414] "containerMap: removing container" podUID="1b84ce8f-b99a-42bd-b684-ae26a52e8b3f" containerName="main"
Oct 31 13:57:52.721562 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.721553    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b84ce8f-b99a-42bd-b684-ae26a52e8b3f" containerName="main"
Oct 31 13:57:52.721650 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.721641    4660 cpu_manager.go:414] "containerMap: removing container" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main"
Oct 31 13:57:52.721718 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.721709    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main"
Oct 31 13:57:52.721811 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.721800    4660 cpu_manager.go:414] "containerMap: removing container" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main"
Oct 31 13:57:52.721887 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.721878    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main"
Oct 31 13:57:52.721976 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.721967    4660 cpu_manager.go:414] "containerMap: removing container" podUID="cec915fa-dbf8-43f5-bb74-db141931047c" containerName="maas-api"
Oct 31 13:57:52.722046 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.722037    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec915fa-dbf8-43f5-bb74-db141931047c" containerName="maas-api"
Oct 31 13:57:52.722321 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.722308    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6bad0c4-bf7f-4f21-a957-8014c12c3aba" containerName="main"
Oct 31 13:57:52.722419 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.722410    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b84ce8f-b99a-42bd-b684-ae26a52e8b3f" containerName="main"
Oct 31 13:57:52.722499 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.722490    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8742e18-cae9-470f-8279-a78c237a4c58" containerName="main"
Oct 31 13:57:52.722571 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.722563    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="20570d2d-2c07-487b-a561-59e10e351154" containerName="main"
Oct 31 13:57:52.722645 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.722637    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec915fa-dbf8-43f5-bb74-db141931047c" containerName="maas-api"
Oct 31 13:57:52.722713 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.722704    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa53f5f-46bc-443c-a9ad-4382eec12895" containerName="istio-proxy"
Oct 31 13:57:52.722806 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.722783    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb2b8e0-12fa-4b14-ac14-75e2aade9b80" containerName="main"
Oct 31 13:57:52.739359 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.739282    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"]
Oct 31 13:57:52.739672 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.739653    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"
Oct 31 13:57:52.810437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.804230    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5af6eb\""
Oct 31 13:57:52.810437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.804435    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5af6eb-dockercfg-ltcsg\""
Oct 31 13:57:52.960315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.954452    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/511009eb-000b-462f-a849-2f4edcacaf3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws\" (UID: \"511009eb-000b-462f-a849-2f4edcacaf3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"
Oct 31 13:57:52.960315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:52.954535    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbxtk\" (UniqueName: \"kubernetes.io/projected/511009eb-000b-462f-a849-2f4edcacaf3e-kube-api-access-kbxtk\") pod \"payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws\" (UID: \"511009eb-000b-462f-a849-2f4edcacaf3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"
Oct 31 13:57:53.068500 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:53.066728    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/511009eb-000b-462f-a849-2f4edcacaf3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws\" (UID: \"511009eb-000b-462f-a849-2f4edcacaf3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"
Oct 31 13:57:53.068500 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:53.066817    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbxtk\" (UniqueName: \"kubernetes.io/projected/511009eb-000b-462f-a849-2f4edcacaf3e-kube-api-access-kbxtk\") pod \"payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws\" (UID: \"511009eb-000b-462f-a849-2f4edcacaf3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"
Oct 31 13:57:53.068500 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:53.068324    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/511009eb-000b-462f-a849-2f4edcacaf3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws\" (UID: \"511009eb-000b-462f-a849-2f4edcacaf3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"
Oct 31 13:57:53.118318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:53.117116    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbxtk\" (UniqueName: \"kubernetes.io/projected/511009eb-000b-462f-a849-2f4edcacaf3e-kube-api-access-kbxtk\") pod \"payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws\" (UID: \"511009eb-000b-462f-a849-2f4edcacaf3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"
Oct 31 13:57:53.371842 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:53.371799    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"
Oct 31 13:57:53.664581 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:57:53.664545    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511009eb_000b_462f_a849_2f4edcacaf3e.slice/crio-194a7235ba4bacf786509464b237b0605f745b07afec2f5149bf5c3046624b48 WatchSource:0}: Error finding container 194a7235ba4bacf786509464b237b0605f745b07afec2f5149bf5c3046624b48: Status 404 returned error can't find the container with id 194a7235ba4bacf786509464b237b0605f745b07afec2f5149bf5c3046624b48
Oct 31 13:57:53.664939 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:53.664870    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"]
Oct 31 13:57:54.066975 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:54.066882    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws" event={"ID":"511009eb-000b-462f-a849-2f4edcacaf3e","Type":"ContainerStarted","Data":"187b83d059a6151478e38c21f98e36462660f4bcdd59d3ef2879143865ce370c"}
Oct 31 13:57:54.066975 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:54.066933    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws" event={"ID":"511009eb-000b-462f-a849-2f4edcacaf3e","Type":"ContainerStarted","Data":"194a7235ba4bacf786509464b237b0605f745b07afec2f5149bf5c3046624b48"}
Oct 31 13:57:54.067217 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:54.067029    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"
Oct 31 13:57:54.089205 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:57:54.089144    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws" podStartSLOduration=2.089123334 podStartE2EDuration="2.089123334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:54.087482194 +0000 UTC m=+3273.706141358" watchObservedRunningTime="2026-10-31 13:57:54.089123334 +0000 UTC m=+3273.707782498"
Oct 31 13:58:04.843437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:04.840689    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"]
Oct 31 13:58:04.843437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:04.841000    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv" podUID="90939bcc-e6bc-42f3-b535-5d10ec96b1be" containerName="maas-api" containerID="cri-o://baadc5c8f41f0c200fd7a14bb243f3c41d78b05e852225fec0eb3e3abe54c613" gracePeriod=30
Oct 31 13:58:05.104372 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:05.103517    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"
Oct 31 13:58:05.547315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:05.542783    4660 generic.go:356] "Generic (PLEG): container finished" podID="90939bcc-e6bc-42f3-b535-5d10ec96b1be" containerID="baadc5c8f41f0c200fd7a14bb243f3c41d78b05e852225fec0eb3e3abe54c613" exitCode=0
Oct 31 13:58:05.547315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:05.542838    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv" event={"ID":"90939bcc-e6bc-42f3-b535-5d10ec96b1be","Type":"ContainerDied","Data":"baadc5c8f41f0c200fd7a14bb243f3c41d78b05e852225fec0eb3e3abe54c613"}
Oct 31 13:58:05.769715 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:05.769689    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:58:05.839075 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:05.838980    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90939bcc-e6bc-42f3-b535-5d10ec96b1be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90939bcc-e6bc-42f3-b535-5d10ec96b1be-metrics-certs\") pod \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\" (UID: \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\") "
Oct 31 13:58:05.839260 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:05.839159    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90939bcc-e6bc-42f3-b535-5d10ec96b1be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90939bcc-e6bc-42f3-b535-5d10ec96b1be-maas-api-tls\") pod \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\" (UID: \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\") "
Oct 31 13:58:05.839260 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:05.839241    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90939bcc-e6bc-42f3-b535-5d10ec96b1be-kube-api-access-zps47\" (UniqueName: \"kubernetes.io/projected/90939bcc-e6bc-42f3-b535-5d10ec96b1be-kube-api-access-zps47\") pod \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\" (UID: \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\") "
Oct 31 13:58:05.839691 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:05.839656    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90939bcc-e6bc-42f3-b535-5d10ec96b1be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90939bcc-e6bc-42f3-b535-5d10ec96b1be-openshift-service-ca\") pod \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\" (UID: \"90939bcc-e6bc-42f3-b535-5d10ec96b1be\") "
Oct 31 13:58:05.840881 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:05.840845    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90939bcc-e6bc-42f3-b535-5d10ec96b1be-openshift-service-ca" pod "90939bcc-e6bc-42f3-b535-5d10ec96b1be" (UID: "90939bcc-e6bc-42f3-b535-5d10ec96b1be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:58:05.847714 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:05.847685    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90939bcc-e6bc-42f3-b535-5d10ec96b1be-kube-api-access-zps47" pod "90939bcc-e6bc-42f3-b535-5d10ec96b1be" (UID: "90939bcc-e6bc-42f3-b535-5d10ec96b1be"). InnerVolumeSpecName "kube-api-access-zps47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:05.853772 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:05.853713    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90939bcc-e6bc-42f3-b535-5d10ec96b1be-metrics-certs" pod "90939bcc-e6bc-42f3-b535-5d10ec96b1be" (UID: "90939bcc-e6bc-42f3-b535-5d10ec96b1be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:58:05.856463 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:05.856433    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90939bcc-e6bc-42f3-b535-5d10ec96b1be-maas-api-tls" pod "90939bcc-e6bc-42f3-b535-5d10ec96b1be" (UID: "90939bcc-e6bc-42f3-b535-5d10ec96b1be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:58:05.942792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:05.942220    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90939bcc-e6bc-42f3-b535-5d10ec96b1be-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:05.942792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:05.942262    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90939bcc-e6bc-42f3-b535-5d10ec96b1be-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:05.942792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:05.942277    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90939bcc-e6bc-42f3-b535-5d10ec96b1be-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:05.942792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:05.942305    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zps47\" (UniqueName: \"kubernetes.io/projected/90939bcc-e6bc-42f3-b535-5d10ec96b1be-kube-api-access-zps47\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:06.552597 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:06.552569    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"
Oct 31 13:58:06.552734 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:06.552587    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv" event={"ID":"90939bcc-e6bc-42f3-b535-5d10ec96b1be","Type":"ContainerDied","Data":"82ffc7fa2e80bedd86e709c78f25ed6f840984f9488f65b67578db8f667bb46d"}
Oct 31 13:58:06.552734 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:06.552652    4660 scope.go:122] "RemoveContainer" containerID="baadc5c8f41f0c200fd7a14bb243f3c41d78b05e852225fec0eb3e3abe54c613"
Oct 31 13:58:06.575062 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:06.575036    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"]
Oct 31 13:58:06.582918 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:06.582891    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5286ea-7877755f7b-htfgv"]
Oct 31 13:58:08.560315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:08.559774    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90939bcc-e6bc-42f3-b535-5d10ec96b1be" path="/var/lib/kubelet/pods/90939bcc-e6bc-42f3-b535-5d10ec96b1be/volumes"
Oct 31 13:58:13.261010 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.260976    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"]
Oct 31 13:58:13.262865 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.262826    4660 cpu_manager.go:414] "containerMap: removing container" podUID="90939bcc-e6bc-42f3-b535-5d10ec96b1be" containerName="maas-api"
Oct 31 13:58:13.262865 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.262858    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90939bcc-e6bc-42f3-b535-5d10ec96b1be" containerName="maas-api"
Oct 31 13:58:13.263119 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.263100    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="90939bcc-e6bc-42f3-b535-5d10ec96b1be" containerName="maas-api"
Oct 31 13:58:13.271328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.269923    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 13:58:13.279309 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.279152    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5af6eb-serving-cert\""
Oct 31 13:58:13.284817 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.284796    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"]
Oct 31 13:58:13.290223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.290195    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"]
Oct 31 13:58:13.290553 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.290497    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" containerID="cri-o://1dbb076eaceb11cedc9d12913ee07374dddbb8926f27fa60586c47e11014253f" gracePeriod=30
Oct 31 13:58:13.292138 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.292077    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f426a22f-701e-46ba-8a6b-99e13ceebfb5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578\" (UID: \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 13:58:13.292271 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.292250    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f426a22f-701e-46ba-8a6b-99e13ceebfb5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578\" (UID: \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 13:58:13.292423 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.292381    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n799z\" (UniqueName: \"kubernetes.io/projected/f426a22f-701e-46ba-8a6b-99e13ceebfb5-kube-api-access-n799z\") pod \"maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578\" (UID: \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 13:58:13.292468 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.292451    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f426a22f-701e-46ba-8a6b-99e13ceebfb5-metrics-certs\") pod \"maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578\" (UID: \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 13:58:13.393638 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.393600    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f426a22f-701e-46ba-8a6b-99e13ceebfb5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578\" (UID: \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 13:58:13.393818 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.393729    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f426a22f-701e-46ba-8a6b-99e13ceebfb5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578\" (UID: \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 13:58:13.393818 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.393774    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n799z\" (UniqueName: \"kubernetes.io/projected/f426a22f-701e-46ba-8a6b-99e13ceebfb5-kube-api-access-n799z\") pod \"maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578\" (UID: \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 13:58:13.393960 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.393859    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f426a22f-701e-46ba-8a6b-99e13ceebfb5-metrics-certs\") pod \"maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578\" (UID: \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 13:58:13.394525 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.394498    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f426a22f-701e-46ba-8a6b-99e13ceebfb5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578\" (UID: \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 13:58:13.396108 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.396086    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f426a22f-701e-46ba-8a6b-99e13ceebfb5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578\" (UID: \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 13:58:13.396206 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.396186    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f426a22f-701e-46ba-8a6b-99e13ceebfb5-metrics-certs\") pod \"maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578\" (UID: \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 13:58:13.406427 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.406403    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n799z\" (UniqueName: \"kubernetes.io/projected/f426a22f-701e-46ba-8a6b-99e13ceebfb5-kube-api-access-n799z\") pod \"maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578\" (UID: \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 13:58:13.593375 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:13.592576    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 13:58:14.117383 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:58:14.117341    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf426a22f_701e_46ba_8a6b_99e13ceebfb5.slice/crio-f8e6d6c8462c1f512740569921e0a25a8896fc50c80d72a600440b91244b5cfc WatchSource:0}: Error finding container f8e6d6c8462c1f512740569921e0a25a8896fc50c80d72a600440b91244b5cfc: Status 404 returned error can't find the container with id f8e6d6c8462c1f512740569921e0a25a8896fc50c80d72a600440b91244b5cfc
Oct 31 13:58:14.153333 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:14.149884    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"]
Oct 31 13:58:14.692069 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:14.692038    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578" event={"ID":"f426a22f-701e-46ba-8a6b-99e13ceebfb5","Type":"ContainerStarted","Data":"f8e6d6c8462c1f512740569921e0a25a8896fc50c80d72a600440b91244b5cfc"}
Oct 31 13:58:15.702498 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:15.702451    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578" event={"ID":"f426a22f-701e-46ba-8a6b-99e13ceebfb5","Type":"ContainerStarted","Data":"f7d379f57f1af6977007e6ff1bd4d6e35cb061fb8c71998d0b39a912d12d2502"}
Oct 31 13:58:15.703008 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:15.702733    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 13:58:15.726514 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:15.726445    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578" podStartSLOduration=2.289877783 podStartE2EDuration="2.726428492s" totalImagesPullingTime="436.550709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:58:15.723361878 +0000 UTC m=+3295.342021044" watchObservedRunningTime="2026-10-31 13:58:15.726428492 +0000 UTC m=+3295.345087657"
Oct 31 13:58:16.110596 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:16.108494    4660 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:58:16.110596 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:16.108563    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:58:18.667165 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.667105    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:58:18.746236 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.746198    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-workload-socket\") pod \"96a7105e-a896-47cd-b203-feab72f829c3\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") "
Oct 31 13:58:18.746561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.746357    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-credential-socket\") pod \"96a7105e-a896-47cd-b203-feab72f829c3\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") "
Oct 31 13:58:18.746561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.746478    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-istio-data\") pod \"96a7105e-a896-47cd-b203-feab72f829c3\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") "
Oct 31 13:58:18.746561 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.746527    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a7105e-a896-47cd-b203-feab72f829c3-istio-token\" (UniqueName: \"kubernetes.io/projected/96a7105e-a896-47cd-b203-feab72f829c3-istio-token\") pod \"96a7105e-a896-47cd-b203-feab72f829c3\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") "
Oct 31 13:58:18.746762 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.746727    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-workload-certs\") pod \"96a7105e-a896-47cd-b203-feab72f829c3\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") "
Oct 31 13:58:18.746992 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.746969    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96a7105e-a896-47cd-b203-feab72f829c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a7105e-a896-47cd-b203-feab72f829c3-istio-podinfo\") pod \"96a7105e-a896-47cd-b203-feab72f829c3\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") "
Oct 31 13:58:18.747113 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.747093    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a7105e-a896-47cd-b203-feab72f829c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a7105e-a896-47cd-b203-feab72f829c3-istiod-ca-cert\") pod \"96a7105e-a896-47cd-b203-feab72f829c3\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") "
Oct 31 13:58:18.747192 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.747137    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-istio-envoy\") pod \"96a7105e-a896-47cd-b203-feab72f829c3\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") "
Oct 31 13:58:18.747192 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.746976    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-workload-socket" pod "96a7105e-a896-47cd-b203-feab72f829c3" (UID: "96a7105e-a896-47cd-b203-feab72f829c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:18.747192 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.747182    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a7105e-a896-47cd-b203-feab72f829c3-kube-api-access-bm25x\" (UniqueName: \"kubernetes.io/projected/96a7105e-a896-47cd-b203-feab72f829c3-kube-api-access-bm25x\") pod \"96a7105e-a896-47cd-b203-feab72f829c3\" (UID: \"96a7105e-a896-47cd-b203-feab72f829c3\") "
Oct 31 13:58:18.747192 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.747035    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-istio-data" pod "96a7105e-a896-47cd-b203-feab72f829c3" (UID: "96a7105e-a896-47cd-b203-feab72f829c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:18.747798 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.747745    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-workload-certs" pod "96a7105e-a896-47cd-b203-feab72f829c3" (UID: "96a7105e-a896-47cd-b203-feab72f829c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:18.747798 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.747762    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-credential-socket" pod "96a7105e-a896-47cd-b203-feab72f829c3" (UID: "96a7105e-a896-47cd-b203-feab72f829c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:18.747943 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.747918    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:18.747995 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.747939    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:18.747995 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.747955    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:18.747995 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.747969    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:18.748629 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.748604    4660 generic.go:356] "Generic (PLEG): container finished" podID="96a7105e-a896-47cd-b203-feab72f829c3" containerID="1dbb076eaceb11cedc9d12913ee07374dddbb8926f27fa60586c47e11014253f" exitCode=0
Oct 31 13:58:18.749352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.748666    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"
Oct 31 13:58:18.749352 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.748686    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" event={"ID":"96a7105e-a896-47cd-b203-feab72f829c3","Type":"ContainerDied","Data":"1dbb076eaceb11cedc9d12913ee07374dddbb8926f27fa60586c47e11014253f"}
Oct 31 13:58:18.749675 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.749652    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a7105e-a896-47cd-b203-feab72f829c3-istiod-ca-cert" pod "96a7105e-a896-47cd-b203-feab72f829c3" (UID: "96a7105e-a896-47cd-b203-feab72f829c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:58:18.751439 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.751411    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm" event={"ID":"96a7105e-a896-47cd-b203-feab72f829c3","Type":"ContainerDied","Data":"b5c6b6ccc13aadff9429ee599e0eddf6fff6349f2925eabf8017d9b8375deddb"}
Oct 31 13:58:18.751553 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.751462    4660 scope.go:122] "RemoveContainer" containerID="1dbb076eaceb11cedc9d12913ee07374dddbb8926f27fa60586c47e11014253f"
Oct 31 13:58:18.752430 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.752405    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a7105e-a896-47cd-b203-feab72f829c3-kube-api-access-bm25x" pod "96a7105e-a896-47cd-b203-feab72f829c3" (UID: "96a7105e-a896-47cd-b203-feab72f829c3"). InnerVolumeSpecName "kube-api-access-bm25x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:18.753228 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.753023    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-istio-envoy" pod "96a7105e-a896-47cd-b203-feab72f829c3" (UID: "96a7105e-a896-47cd-b203-feab72f829c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:18.753228 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.753168    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96a7105e-a896-47cd-b203-feab72f829c3-istio-podinfo" pod "96a7105e-a896-47cd-b203-feab72f829c3" (UID: "96a7105e-a896-47cd-b203-feab72f829c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:58:18.754697 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.754670    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a7105e-a896-47cd-b203-feab72f829c3-istio-token" pod "96a7105e-a896-47cd-b203-feab72f829c3" (UID: "96a7105e-a896-47cd-b203-feab72f829c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:18.785617 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.785584    4660 scope.go:122] "RemoveContainer" containerID="1dbb076eaceb11cedc9d12913ee07374dddbb8926f27fa60586c47e11014253f"
Oct 31 13:58:18.785988 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:58:18.785960    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dbb076eaceb11cedc9d12913ee07374dddbb8926f27fa60586c47e11014253f\": container with ID starting with 1dbb076eaceb11cedc9d12913ee07374dddbb8926f27fa60586c47e11014253f not found: ID does not exist" containerID="1dbb076eaceb11cedc9d12913ee07374dddbb8926f27fa60586c47e11014253f"
Oct 31 13:58:18.786077 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.786000    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbb076eaceb11cedc9d12913ee07374dddbb8926f27fa60586c47e11014253f"} err="failed to get container status \"1dbb076eaceb11cedc9d12913ee07374dddbb8926f27fa60586c47e11014253f\": rpc error: code = NotFound desc = could not find container \"1dbb076eaceb11cedc9d12913ee07374dddbb8926f27fa60586c47e11014253f\": container with ID starting with 1dbb076eaceb11cedc9d12913ee07374dddbb8926f27fa60586c47e11014253f not found: ID does not exist"
Oct 31 13:58:18.849792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.849758    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a7105e-a896-47cd-b203-feab72f829c3-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:18.849792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.849788    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a7105e-a896-47cd-b203-feab72f829c3-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:18.849792 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.849798    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a7105e-a896-47cd-b203-feab72f829c3-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:18.850089 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.849807    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm25x\" (UniqueName: \"kubernetes.io/projected/96a7105e-a896-47cd-b203-feab72f829c3-kube-api-access-bm25x\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:18.850089 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:18.849818    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a7105e-a896-47cd-b203-feab72f829c3-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:19.083414 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:19.083371    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"]
Oct 31 13:58:19.120315 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:19.098731    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hkngm"]
Oct 31 13:58:20.562980 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:20.562886    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a7105e-a896-47cd-b203-feab72f829c3" path="/var/lib/kubelet/pods/96a7105e-a896-47cd-b203-feab72f829c3/volumes"
Oct 31 13:58:20.563490 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:20.563387    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"]
Oct 31 13:58:20.563621 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:20.563596    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx" podUID="acc4f15d-78ea-4803-9d9d-ad445710d934" containerName="istio-proxy" containerID="cri-o://65ee145305b21d0093440be208eb34022bbaeedc3f41c2f061360b10169e71ee" gracePeriod=30
Oct 31 13:58:20.849619 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:20.849520    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:58:20.855417 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:20.855378    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx" podUID="acc4f15d-78ea-4803-9d9d-ad445710d934" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:58:20.924389 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:58:20.924347    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:58:21.334317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:21.334194    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:58:21.339119 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:21.339091    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 13:58:21.342992 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:21.342962    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:58:21.346457 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:21.346433    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 13:58:21.352694 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:21.352669    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:58:21.355709 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:21.355691    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 13:58:21.719609 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:21.719562    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 13:58:25.874271 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:25.873034    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:58:25.876211 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:25.876178    4660 generic.go:356] "Generic (PLEG): container finished" podID="acc4f15d-78ea-4803-9d9d-ad445710d934" containerID="65ee145305b21d0093440be208eb34022bbaeedc3f41c2f061360b10169e71ee" exitCode=0
Oct 31 13:58:25.876419 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:25.876225    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx" event={"ID":"acc4f15d-78ea-4803-9d9d-ad445710d934","Type":"ContainerDied","Data":"65ee145305b21d0093440be208eb34022bbaeedc3f41c2f061360b10169e71ee"}
Oct 31 13:58:25.876419 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:25.876241    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"
Oct 31 13:58:25.876419 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:25.876328    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx" event={"ID":"acc4f15d-78ea-4803-9d9d-ad445710d934","Type":"ContainerDied","Data":"8a2eb533b513651c78b378b138b9853ad30d31a832585c9196e88686763930e4"}
Oct 31 13:58:25.876419 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:25.876372    4660 scope.go:122] "RemoveContainer" containerID="65ee145305b21d0093440be208eb34022bbaeedc3f41c2f061360b10169e71ee"
Oct 31 13:58:25.889951 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:25.889916    4660 scope.go:122] "RemoveContainer" containerID="10cc3753da99e65c253ce224644c71ce661f1539bda8f6970036545aa220ac2a"
Oct 31 13:58:25.903466 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:25.903277    4660 scope.go:122] "RemoveContainer" containerID="65ee145305b21d0093440be208eb34022bbaeedc3f41c2f061360b10169e71ee"
Oct 31 13:58:25.903601 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:58:25.903568    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ee145305b21d0093440be208eb34022bbaeedc3f41c2f061360b10169e71ee\": container with ID starting with 65ee145305b21d0093440be208eb34022bbaeedc3f41c2f061360b10169e71ee not found: ID does not exist" containerID="65ee145305b21d0093440be208eb34022bbaeedc3f41c2f061360b10169e71ee"
Oct 31 13:58:25.903706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:25.903600    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ee145305b21d0093440be208eb34022bbaeedc3f41c2f061360b10169e71ee"} err="failed to get container status \"65ee145305b21d0093440be208eb34022bbaeedc3f41c2f061360b10169e71ee\": rpc error: code = NotFound desc = could not find container \"65ee145305b21d0093440be208eb34022bbaeedc3f41c2f061360b10169e71ee\": container with ID starting with 65ee145305b21d0093440be208eb34022bbaeedc3f41c2f061360b10169e71ee not found: ID does not exist"
Oct 31 13:58:25.903706 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:25.903623    4660 scope.go:122] "RemoveContainer" containerID="10cc3753da99e65c253ce224644c71ce661f1539bda8f6970036545aa220ac2a"
Oct 31 13:58:25.904004 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:58:25.903855    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10cc3753da99e65c253ce224644c71ce661f1539bda8f6970036545aa220ac2a\": container with ID starting with 10cc3753da99e65c253ce224644c71ce661f1539bda8f6970036545aa220ac2a not found: ID does not exist" containerID="10cc3753da99e65c253ce224644c71ce661f1539bda8f6970036545aa220ac2a"
Oct 31 13:58:25.904004 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:25.903938    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10cc3753da99e65c253ce224644c71ce661f1539bda8f6970036545aa220ac2a"} err="failed to get container status \"10cc3753da99e65c253ce224644c71ce661f1539bda8f6970036545aa220ac2a\": rpc error: code = NotFound desc = could not find container \"10cc3753da99e65c253ce224644c71ce661f1539bda8f6970036545aa220ac2a\": container with ID starting with 10cc3753da99e65c253ce224644c71ce661f1539bda8f6970036545aa220ac2a not found: ID does not exist"
Oct 31 13:58:26.009565 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.009529    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc4f15d-78ea-4803-9d9d-ad445710d934-kube-api-access-8cnx4\" (UniqueName: \"kubernetes.io/projected/acc4f15d-78ea-4803-9d9d-ad445710d934-kube-api-access-8cnx4\") pod \"acc4f15d-78ea-4803-9d9d-ad445710d934\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") "
Oct 31 13:58:26.009745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.009594    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-podinfo\") pod \"acc4f15d-78ea-4803-9d9d-ad445710d934\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") "
Oct 31 13:58:26.009745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.009619    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-data\") pod \"acc4f15d-78ea-4803-9d9d-ad445710d934\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") "
Oct 31 13:58:26.009745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.009649    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc4f15d-78ea-4803-9d9d-ad445710d934-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc4f15d-78ea-4803-9d9d-ad445710d934-istiod-ca-cert\") pod \"acc4f15d-78ea-4803-9d9d-ad445710d934\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") "
Oct 31 13:58:26.009745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.009679    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-credential-socket\") pod \"acc4f15d-78ea-4803-9d9d-ad445710d934\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") "
Oct 31 13:58:26.009745 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.009714    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-workload-certs\") pod \"acc4f15d-78ea-4803-9d9d-ad445710d934\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") "
Oct 31 13:58:26.010085 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.009816    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-workload-socket\") pod \"acc4f15d-78ea-4803-9d9d-ad445710d934\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") "
Oct 31 13:58:26.010085 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.009893    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-token\" (UniqueName: \"kubernetes.io/projected/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-token\") pod \"acc4f15d-78ea-4803-9d9d-ad445710d934\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") "
Oct 31 13:58:26.010085 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.009944    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-envoy\") pod \"acc4f15d-78ea-4803-9d9d-ad445710d934\" (UID: \"acc4f15d-78ea-4803-9d9d-ad445710d934\") "
Oct 31 13:58:26.010085 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.010042    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-credential-socket" pod "acc4f15d-78ea-4803-9d9d-ad445710d934" (UID: "acc4f15d-78ea-4803-9d9d-ad445710d934"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:26.010329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.010096    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-workload-certs" pod "acc4f15d-78ea-4803-9d9d-ad445710d934" (UID: "acc4f15d-78ea-4803-9d9d-ad445710d934"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:26.010401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.010320    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-data" pod "acc4f15d-78ea-4803-9d9d-ad445710d934" (UID: "acc4f15d-78ea-4803-9d9d-ad445710d934"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:26.010401 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.010322    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc4f15d-78ea-4803-9d9d-ad445710d934-istiod-ca-cert" pod "acc4f15d-78ea-4803-9d9d-ad445710d934" (UID: "acc4f15d-78ea-4803-9d9d-ad445710d934"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:58:26.010515 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.010395    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-workload-socket" pod "acc4f15d-78ea-4803-9d9d-ad445710d934" (UID: "acc4f15d-78ea-4803-9d9d-ad445710d934"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:26.010854 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.010817    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:26.010854 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.010845    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc4f15d-78ea-4803-9d9d-ad445710d934-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:26.010985 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.010861    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:26.010985 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.010875    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:26.010985 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.010888    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:26.012730 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.012704    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-podinfo" pod "acc4f15d-78ea-4803-9d9d-ad445710d934" (UID: "acc4f15d-78ea-4803-9d9d-ad445710d934"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:58:26.012730 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.012723    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-token" pod "acc4f15d-78ea-4803-9d9d-ad445710d934" (UID: "acc4f15d-78ea-4803-9d9d-ad445710d934"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:26.012730 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.012706    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc4f15d-78ea-4803-9d9d-ad445710d934-kube-api-access-8cnx4" pod "acc4f15d-78ea-4803-9d9d-ad445710d934" (UID: "acc4f15d-78ea-4803-9d9d-ad445710d934"). InnerVolumeSpecName "kube-api-access-8cnx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:26.012962 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.012752    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-envoy" pod "acc4f15d-78ea-4803-9d9d-ad445710d934" (UID: "acc4f15d-78ea-4803-9d9d-ad445710d934"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:26.112036 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.112000    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cnx4\" (UniqueName: \"kubernetes.io/projected/acc4f15d-78ea-4803-9d9d-ad445710d934-kube-api-access-8cnx4\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:26.112036 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.112028    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:26.112036 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.112040    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:26.113712 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.112051    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc4f15d-78ea-4803-9d9d-ad445710d934-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:26.209149 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.209111    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"]
Oct 31 13:58:26.214017 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.213985    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-d86kx"]
Oct 31 13:58:26.531277 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:26.531194    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc4f15d-78ea-4803-9d9d-ad445710d934" path="/var/lib/kubelet/pods/acc4f15d-78ea-4803-9d9d-ad445710d934/volumes"
Oct 31 13:58:37.105614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:37.105574    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"]
Oct 31 13:58:42.106357 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:42.106288    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws" podUID="511009eb-000b-462f-a849-2f4edcacaf3e" containerName="payload-processing" containerID="cri-o://187b83d059a6151478e38c21f98e36462660f4bcdd59d3ef2879143865ce370c" gracePeriod=25
Oct 31 13:58:42.470711 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:42.470675    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"
Oct 31 13:58:42.579730 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:42.579687    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/511009eb-000b-462f-a849-2f4edcacaf3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/511009eb-000b-462f-a849-2f4edcacaf3e-plugins-config-volume\") pod \"511009eb-000b-462f-a849-2f4edcacaf3e\" (UID: \"511009eb-000b-462f-a849-2f4edcacaf3e\") "
Oct 31 13:58:42.579921 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:42.579910    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/511009eb-000b-462f-a849-2f4edcacaf3e-kube-api-access-kbxtk\" (UniqueName: \"kubernetes.io/projected/511009eb-000b-462f-a849-2f4edcacaf3e-kube-api-access-kbxtk\") pod \"511009eb-000b-462f-a849-2f4edcacaf3e\" (UID: \"511009eb-000b-462f-a849-2f4edcacaf3e\") "
Oct 31 13:58:42.580176 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:42.580139    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511009eb-000b-462f-a849-2f4edcacaf3e-plugins-config-volume" pod "511009eb-000b-462f-a849-2f4edcacaf3e" (UID: "511009eb-000b-462f-a849-2f4edcacaf3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:58:42.580438 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:42.580418    4660 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/511009eb-000b-462f-a849-2f4edcacaf3e-plugins-config-volume\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:42.582126 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:42.582104    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511009eb-000b-462f-a849-2f4edcacaf3e-kube-api-access-kbxtk" pod "511009eb-000b-462f-a849-2f4edcacaf3e" (UID: "511009eb-000b-462f-a849-2f4edcacaf3e"). InnerVolumeSpecName "kube-api-access-kbxtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:42.681793 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:42.681682    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbxtk\" (UniqueName: \"kubernetes.io/projected/511009eb-000b-462f-a849-2f4edcacaf3e-kube-api-access-kbxtk\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:43.086433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:43.086340    4660 generic.go:356] "Generic (PLEG): container finished" podID="511009eb-000b-462f-a849-2f4edcacaf3e" containerID="187b83d059a6151478e38c21f98e36462660f4bcdd59d3ef2879143865ce370c" exitCode=0
Oct 31 13:58:43.086433 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:43.086399    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"
Oct 31 13:58:43.086676 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:43.086431    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws" event={"ID":"511009eb-000b-462f-a849-2f4edcacaf3e","Type":"ContainerDied","Data":"187b83d059a6151478e38c21f98e36462660f4bcdd59d3ef2879143865ce370c"}
Oct 31 13:58:43.086676 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:43.086497    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws" event={"ID":"511009eb-000b-462f-a849-2f4edcacaf3e","Type":"ContainerDied","Data":"194a7235ba4bacf786509464b237b0605f745b07afec2f5149bf5c3046624b48"}
Oct 31 13:58:43.086676 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:43.086521    4660 scope.go:122] "RemoveContainer" containerID="187b83d059a6151478e38c21f98e36462660f4bcdd59d3ef2879143865ce370c"
Oct 31 13:58:43.098429 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:43.098403    4660 scope.go:122] "RemoveContainer" containerID="187b83d059a6151478e38c21f98e36462660f4bcdd59d3ef2879143865ce370c"
Oct 31 13:58:43.098709 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:58:43.098687    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"187b83d059a6151478e38c21f98e36462660f4bcdd59d3ef2879143865ce370c\": container with ID starting with 187b83d059a6151478e38c21f98e36462660f4bcdd59d3ef2879143865ce370c not found: ID does not exist" containerID="187b83d059a6151478e38c21f98e36462660f4bcdd59d3ef2879143865ce370c"
Oct 31 13:58:43.098763 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:43.098718    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"187b83d059a6151478e38c21f98e36462660f4bcdd59d3ef2879143865ce370c"} err="failed to get container status \"187b83d059a6151478e38c21f98e36462660f4bcdd59d3ef2879143865ce370c\": rpc error: code = NotFound desc = could not find container \"187b83d059a6151478e38c21f98e36462660f4bcdd59d3ef2879143865ce370c\": container with ID starting with 187b83d059a6151478e38c21f98e36462660f4bcdd59d3ef2879143865ce370c not found: ID does not exist"
Oct 31 13:58:43.114216 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:43.114168    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"]
Oct 31 13:58:43.118396 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:43.118364    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-698d589597-km9ws"]
Oct 31 13:58:44.531931 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:44.531892    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="511009eb-000b-462f-a849-2f4edcacaf3e" path="/var/lib/kubelet/pods/511009eb-000b-462f-a849-2f4edcacaf3e/volumes"
Oct 31 13:58:50.737817 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:58:50.737774    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:58:55.343716 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:55.343678    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"]
Oct 31 13:58:55.355144 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:58:55.355109    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"]
Oct 31 13:59:09.247472 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.247436    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"]
Oct 31 13:59:09.248397 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.248375    4660 cpu_manager.go:414] "containerMap: removing container" podUID="511009eb-000b-462f-a849-2f4edcacaf3e" containerName="payload-processing"
Oct 31 13:59:09.248453 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.248400    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="511009eb-000b-462f-a849-2f4edcacaf3e" containerName="payload-processing"
Oct 31 13:59:09.248453 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.248426    4660 cpu_manager.go:414] "containerMap: removing container" podUID="acc4f15d-78ea-4803-9d9d-ad445710d934" containerName="istio-proxy"
Oct 31 13:59:09.248453 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.248433    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc4f15d-78ea-4803-9d9d-ad445710d934" containerName="istio-proxy"
Oct 31 13:59:09.248453 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.248451    4660 cpu_manager.go:414] "containerMap: removing container" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy"
Oct 31 13:59:09.248624 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.248459    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy"
Oct 31 13:59:09.248659 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.248640    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc4f15d-78ea-4803-9d9d-ad445710d934" containerName="istio-proxy"
Oct 31 13:59:09.248693 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.248658    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="511009eb-000b-462f-a849-2f4edcacaf3e" containerName="payload-processing"
Oct 31 13:59:09.248693 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.248670    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc4f15d-78ea-4803-9d9d-ad445710d934" containerName="istio-proxy"
Oct 31 13:59:09.248693 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.248679    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a7105e-a896-47cd-b203-feab72f829c3" containerName="istio-proxy"
Oct 31 13:59:09.252929 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.252907    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"
Oct 31 13:59:09.260562 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.260536    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f9ca72\""
Oct 31 13:59:09.260705 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.260668    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f9ca72-dockercfg-hvzgx\""
Oct 31 13:59:09.261329 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.261307    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"]
Oct 31 13:59:09.483328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.381237    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85gsl\" (UniqueName: \"kubernetes.io/projected/4afbe3c3-aee3-4ed6-a37b-95e3c96c1792-kube-api-access-85gsl\") pod \"payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv\" (UID: \"4afbe3c3-aee3-4ed6-a37b-95e3c96c1792\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"
Oct 31 13:59:09.483328 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.381406    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afbe3c3-aee3-4ed6-a37b-95e3c96c1792-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv\" (UID: \"4afbe3c3-aee3-4ed6-a37b-95e3c96c1792\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"
Oct 31 13:59:09.568257 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.568141    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afbe3c3-aee3-4ed6-a37b-95e3c96c1792-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv\" (UID: \"4afbe3c3-aee3-4ed6-a37b-95e3c96c1792\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"
Oct 31 13:59:09.580478 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.580437    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85gsl\" (UniqueName: \"kubernetes.io/projected/4afbe3c3-aee3-4ed6-a37b-95e3c96c1792-kube-api-access-85gsl\") pod \"payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv\" (UID: \"4afbe3c3-aee3-4ed6-a37b-95e3c96c1792\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"
Oct 31 13:59:09.631320 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.630502    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afbe3c3-aee3-4ed6-a37b-95e3c96c1792-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv\" (UID: \"4afbe3c3-aee3-4ed6-a37b-95e3c96c1792\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"
Oct 31 13:59:09.667370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.667333    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85gsl\" (UniqueName: \"kubernetes.io/projected/4afbe3c3-aee3-4ed6-a37b-95e3c96c1792-kube-api-access-85gsl\") pod \"payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv\" (UID: \"4afbe3c3-aee3-4ed6-a37b-95e3c96c1792\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"
Oct 31 13:59:09.881653 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:09.881615    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"
Oct 31 13:59:10.183955 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.183917    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"]
Oct 31 13:59:10.184536 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:59:10.184511    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afbe3c3_aee3_4ed6_a37b_95e3c96c1792.slice/crio-1ccaf81d4f987d186e47af6b9f81168d30bf3a82d663ec13824f945d921a5619 WatchSource:0}: Error finding container 1ccaf81d4f987d186e47af6b9f81168d30bf3a82d663ec13824f945d921a5619: Status 404 returned error can't find the container with id 1ccaf81d4f987d186e47af6b9f81168d30bf3a82d663ec13824f945d921a5619
Oct 31 13:59:10.367155 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.367113    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main" containerID="cri-o://aac5c2cbfc08786fd20489700311a152f01770c0ebb2ca1caba6242d1c927fad" gracePeriod=45
Oct 31 13:59:10.380421 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.380363    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main" containerID="cri-o://83f935155f328e84a42ed360654630e1c78eca6445bfb9b1266bc9795ef3b02f" gracePeriod=45
Oct 31 13:59:10.460253 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:59:10.460172    4660 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod150de627_a508_4a67_8be3_194452c9325d.slice/crio-conmon-83f935155f328e84a42ed360654630e1c78eca6445bfb9b1266bc9795ef3b02f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod150de627_a508_4a67_8be3_194452c9325d.slice/crio-83f935155f328e84a42ed360654630e1c78eca6445bfb9b1266bc9795ef3b02f.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 13:59:10.623007 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.622975    4660 generic.go:356] "Generic (PLEG): container finished" podID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerID="aac5c2cbfc08786fd20489700311a152f01770c0ebb2ca1caba6242d1c927fad" exitCode=0
Oct 31 13:59:10.623366 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.623343    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" event={"ID":"e64ea0e4-ea86-497c-b24c-f8326ebe340b","Type":"ContainerDied","Data":"aac5c2cbfc08786fd20489700311a152f01770c0ebb2ca1caba6242d1c927fad"}
Oct 31 13:59:10.625731 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.625703    4660 generic.go:356] "Generic (PLEG): container finished" podID="150de627-a508-4a67-8be3-194452c9325d" containerID="83f935155f328e84a42ed360654630e1c78eca6445bfb9b1266bc9795ef3b02f" exitCode=0
Oct 31 13:59:10.626208 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.625903    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" event={"ID":"150de627-a508-4a67-8be3-194452c9325d","Type":"ContainerDied","Data":"83f935155f328e84a42ed360654630e1c78eca6445bfb9b1266bc9795ef3b02f"}
Oct 31 13:59:10.629388 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.629368    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv" event={"ID":"4afbe3c3-aee3-4ed6-a37b-95e3c96c1792","Type":"ContainerStarted","Data":"e4049819458d7f69509125a3acc406cc1226a14907e4a79462a5012f6fdd4b86"}
Oct 31 13:59:10.629600 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.629578    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"
Oct 31 13:59:10.630244 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.630228    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv" event={"ID":"4afbe3c3-aee3-4ed6-a37b-95e3c96c1792","Type":"ContainerStarted","Data":"1ccaf81d4f987d186e47af6b9f81168d30bf3a82d663ec13824f945d921a5619"}
Oct 31 13:59:10.787899 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.787877    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:59:10.831317 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.827648    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv" podStartSLOduration=1.8276275229999999 podStartE2EDuration="1.827627523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:59:10.662848433 +0000 UTC m=+3350.281507610" watchObservedRunningTime="2026-10-31 13:59:10.827627523 +0000 UTC m=+3350.446286691"
Oct 31 13:59:10.855318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.837573    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-model-cache\") pod \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") "
Oct 31 13:59:10.855318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.848189    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-home\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-home\") pod \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") "
Oct 31 13:59:10.855318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.837878    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-model-cache" pod "e64ea0e4-ea86-497c-b24c-f8326ebe340b" (UID: "e64ea0e4-ea86-497c-b24c-f8326ebe340b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:10.855318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.848287    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64ea0e4-ea86-497c-b24c-f8326ebe340b-tls-certs\" (UniqueName: \"kubernetes.io/secret/e64ea0e4-ea86-497c-b24c-f8326ebe340b-tls-certs\") pod \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") "
Oct 31 13:59:10.855318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.848511    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-tmp-dir\") pod \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") "
Oct 31 13:59:10.855318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.848554    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64ea0e4-ea86-497c-b24c-f8326ebe340b-kube-api-access-m5vjn\" (UniqueName: \"kubernetes.io/projected/e64ea0e4-ea86-497c-b24c-f8326ebe340b-kube-api-access-m5vjn\") pod \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") "
Oct 31 13:59:10.855318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.848603    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-dshm\") pod \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\" (UID: \"e64ea0e4-ea86-497c-b24c-f8326ebe340b\") "
Oct 31 13:59:10.855318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.848655    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-home" pod "e64ea0e4-ea86-497c-b24c-f8326ebe340b" (UID: "e64ea0e4-ea86-497c-b24c-f8326ebe340b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:10.855318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.848973    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-tmp-dir" pod "e64ea0e4-ea86-497c-b24c-f8326ebe340b" (UID: "e64ea0e4-ea86-497c-b24c-f8326ebe340b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:10.855318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.849354    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:10.855318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.849377    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:10.855318 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.849393    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:10.857538 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.857075    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:59:10.859123 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.859097    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-dshm" pod "e64ea0e4-ea86-497c-b24c-f8326ebe340b" (UID: "e64ea0e4-ea86-497c-b24c-f8326ebe340b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:10.859763 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.859726    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64ea0e4-ea86-497c-b24c-f8326ebe340b-tls-certs" pod "e64ea0e4-ea86-497c-b24c-f8326ebe340b" (UID: "e64ea0e4-ea86-497c-b24c-f8326ebe340b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:59:10.859993 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.859967    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64ea0e4-ea86-497c-b24c-f8326ebe340b-kube-api-access-m5vjn" pod "e64ea0e4-ea86-497c-b24c-f8326ebe340b" (UID: "e64ea0e4-ea86-497c-b24c-f8326ebe340b"). InnerVolumeSpecName "kube-api-access-m5vjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:59:10.950316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.949827    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150de627-a508-4a67-8be3-194452c9325d-kube-api-access-jxlxr\" (UniqueName: \"kubernetes.io/projected/150de627-a508-4a67-8be3-194452c9325d-kube-api-access-jxlxr\") pod \"150de627-a508-4a67-8be3-194452c9325d\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") "
Oct 31 13:59:10.950316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.949927    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-model-cache\") pod \"150de627-a508-4a67-8be3-194452c9325d\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") "
Oct 31 13:59:10.950316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.950049    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-tmp-dir\") pod \"150de627-a508-4a67-8be3-194452c9325d\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") "
Oct 31 13:59:10.950316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.950100    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-home\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-home\") pod \"150de627-a508-4a67-8be3-194452c9325d\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") "
Oct 31 13:59:10.950316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.950136    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/150de627-a508-4a67-8be3-194452c9325d-tls-certs\" (UniqueName: \"kubernetes.io/secret/150de627-a508-4a67-8be3-194452c9325d-tls-certs\") pod \"150de627-a508-4a67-8be3-194452c9325d\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") "
Oct 31 13:59:10.950316 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.950193    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-dshm\") pod \"150de627-a508-4a67-8be3-194452c9325d\" (UID: \"150de627-a508-4a67-8be3-194452c9325d\") "
Oct 31 13:59:10.950819 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.950634    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5vjn\" (UniqueName: \"kubernetes.io/projected/e64ea0e4-ea86-497c-b24c-f8326ebe340b-kube-api-access-m5vjn\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:10.950819 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.950663    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64ea0e4-ea86-497c-b24c-f8326ebe340b-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:10.950819 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.950672    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64ea0e4-ea86-497c-b24c-f8326ebe340b-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:10.951396 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.951356    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-home" pod "150de627-a508-4a67-8be3-194452c9325d" (UID: "150de627-a508-4a67-8be3-194452c9325d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:10.951515 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.951395    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-model-cache" pod "150de627-a508-4a67-8be3-194452c9325d" (UID: "150de627-a508-4a67-8be3-194452c9325d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:10.952355 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.952331    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-tmp-dir" pod "150de627-a508-4a67-8be3-194452c9325d" (UID: "150de627-a508-4a67-8be3-194452c9325d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:10.953171 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.953145    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-dshm" pod "150de627-a508-4a67-8be3-194452c9325d" (UID: "150de627-a508-4a67-8be3-194452c9325d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:10.953825 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.953804    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/150de627-a508-4a67-8be3-194452c9325d-tls-certs" pod "150de627-a508-4a67-8be3-194452c9325d" (UID: "150de627-a508-4a67-8be3-194452c9325d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:59:10.954206 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:10.954180    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150de627-a508-4a67-8be3-194452c9325d-kube-api-access-jxlxr" pod "150de627-a508-4a67-8be3-194452c9325d" (UID: "150de627-a508-4a67-8be3-194452c9325d"). InnerVolumeSpecName "kube-api-access-jxlxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:59:11.051811 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:11.051703    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxlxr\" (UniqueName: \"kubernetes.io/projected/150de627-a508-4a67-8be3-194452c9325d-kube-api-access-jxlxr\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:11.051811 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:11.051743    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:11.051811 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:11.051762    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:11.051811 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:11.051775    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:11.051811 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:11.051786    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/150de627-a508-4a67-8be3-194452c9325d-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:11.051811 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:11.051800    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/150de627-a508-4a67-8be3-194452c9325d-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:11.637892 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:11.637863    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"
Oct 31 13:59:11.637892 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:11.637867    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f" event={"ID":"e64ea0e4-ea86-497c-b24c-f8326ebe340b","Type":"ContainerDied","Data":"f110a245e7bc6367fa1e3b507b775882ba7e4feb86dd626631adf264dc793193"}
Oct 31 13:59:11.638471 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:11.637925    4660 scope.go:122] "RemoveContainer" containerID="aac5c2cbfc08786fd20489700311a152f01770c0ebb2ca1caba6242d1c927fad"
Oct 31 13:59:11.640549 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:11.640526    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"
Oct 31 13:59:11.645437 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:11.645412    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs" event={"ID":"150de627-a508-4a67-8be3-194452c9325d","Type":"ContainerDied","Data":"d78ac0abc2cf9d72d1c3e9a619a15e362030026d9ff6dfcf701c8806e497c2c7"}
Oct 31 13:59:11.663767 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:11.663749    4660 scope.go:122] "RemoveContainer" containerID="83f935155f328e84a42ed360654630e1c78eca6445bfb9b1266bc9795ef3b02f"
Oct 31 13:59:11.681526 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:11.681498    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"]
Oct 31 13:59:11.685712 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:11.685689    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/premium-w0-7b3ddd-kserve-5b79dc4f5b-xn2qs"]
Oct 31 13:59:11.697478 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:11.697456    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"]
Oct 31 13:59:11.702102 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:11.702081    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/distinct-w0-7b3ddd-kserve-7d6db44b9f-zd88f"]
Oct 31 13:59:12.537356 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:12.537287    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="150de627-a508-4a67-8be3-194452c9325d" path="/var/lib/kubelet/pods/150de627-a508-4a67-8be3-194452c9325d/volumes"
Oct 31 13:59:12.537996 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:12.537973    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" path="/var/lib/kubelet/pods/e64ea0e4-ea86-497c-b24c-f8326ebe340b/volumes"
Oct 31 13:59:20.733636 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:59:20.733588    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:59:22.655150 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:22.655121    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"
Oct 31 13:59:26.797155 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.797118    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"]
Oct 31 13:59:26.797882 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.797861    4660 cpu_manager.go:414] "containerMap: removing container" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main"
Oct 31 13:59:26.797937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.797882    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main"
Oct 31 13:59:26.797937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.797910    4660 cpu_manager.go:414] "containerMap: removing container" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main"
Oct 31 13:59:26.797937 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.797915    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main"
Oct 31 13:59:26.798078 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.797942    4660 cpu_manager.go:414] "containerMap: removing container" podUID="acc4f15d-78ea-4803-9d9d-ad445710d934" containerName="istio-proxy"
Oct 31 13:59:26.798078 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.797948    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc4f15d-78ea-4803-9d9d-ad445710d934" containerName="istio-proxy"
Oct 31 13:59:26.798078 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.798062    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64ea0e4-ea86-497c-b24c-f8326ebe340b" containerName="main"
Oct 31 13:59:26.798078 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.798073    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="150de627-a508-4a67-8be3-194452c9325d" containerName="main"
Oct 31 13:59:26.803646 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.803622    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 13:59:26.823932 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.823905    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f9ca72-serving-cert\""
Oct 31 13:59:26.827003 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.826412    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"]
Oct 31 13:59:26.856385 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.856348    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f82f37eb-0856-4af8-857b-b855316ee6b0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn\" (UID: \"f82f37eb-0856-4af8-857b-b855316ee6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 13:59:26.856550 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.856440    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f82f37eb-0856-4af8-857b-b855316ee6b0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn\" (UID: \"f82f37eb-0856-4af8-857b-b855316ee6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 13:59:26.856622 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.856546    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82f37eb-0856-4af8-857b-b855316ee6b0-metrics-certs\") pod \"maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn\" (UID: \"f82f37eb-0856-4af8-857b-b855316ee6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 13:59:26.856622 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.856604    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ld8g\" (UniqueName: \"kubernetes.io/projected/f82f37eb-0856-4af8-857b-b855316ee6b0-kube-api-access-7ld8g\") pod \"maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn\" (UID: \"f82f37eb-0856-4af8-857b-b855316ee6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 13:59:26.958277 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.958233    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f82f37eb-0856-4af8-857b-b855316ee6b0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn\" (UID: \"f82f37eb-0856-4af8-857b-b855316ee6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 13:59:26.959647 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.959619    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f82f37eb-0856-4af8-857b-b855316ee6b0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn\" (UID: \"f82f37eb-0856-4af8-857b-b855316ee6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 13:59:26.960327 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.960306    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82f37eb-0856-4af8-857b-b855316ee6b0-metrics-certs\") pod \"maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn\" (UID: \"f82f37eb-0856-4af8-857b-b855316ee6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 13:59:26.960873 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.960853    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ld8g\" (UniqueName: \"kubernetes.io/projected/f82f37eb-0856-4af8-857b-b855316ee6b0-kube-api-access-7ld8g\") pod \"maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn\" (UID: \"f82f37eb-0856-4af8-857b-b855316ee6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 13:59:26.961467 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.959341    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f82f37eb-0856-4af8-857b-b855316ee6b0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn\" (UID: \"f82f37eb-0856-4af8-857b-b855316ee6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 13:59:26.966234 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.966210    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f82f37eb-0856-4af8-857b-b855316ee6b0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn\" (UID: \"f82f37eb-0856-4af8-857b-b855316ee6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 13:59:26.984916 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.969888    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82f37eb-0856-4af8-857b-b855316ee6b0-metrics-certs\") pod \"maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn\" (UID: \"f82f37eb-0856-4af8-857b-b855316ee6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 13:59:26.985967 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:26.985884    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ld8g\" (UniqueName: \"kubernetes.io/projected/f82f37eb-0856-4af8-857b-b855316ee6b0-kube-api-access-7ld8g\") pod \"maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn\" (UID: \"f82f37eb-0856-4af8-857b-b855316ee6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 13:59:27.139993 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:27.139956    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 13:59:27.544459 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:27.544423    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"]
Oct 31 13:59:27.861560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:27.861475    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn" event={"ID":"f82f37eb-0856-4af8-857b-b855316ee6b0","Type":"ContainerStarted","Data":"850c86c321661c26f8967831c5a6c3bc1ecc787a8d715ae119bf87239604b009"}
Oct 31 13:59:28.870593 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:28.870551    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn" event={"ID":"f82f37eb-0856-4af8-857b-b855316ee6b0","Type":"ContainerStarted","Data":"00f5a52eab85ed6cb7e1f4b2b14b8d491d826d19965a4cf0edd1cbe05adb54f8"}
Oct 31 13:59:28.871053 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:28.870659    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 13:59:28.895333 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:28.895266    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn" podStartSLOduration=2.383841593 podStartE2EDuration="2.89525237s" totalImagesPullingTime="511.410777ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:59:28.892801938 +0000 UTC m=+3368.511461103" watchObservedRunningTime="2026-10-31 13:59:28.89525237 +0000 UTC m=+3368.513911533"
Oct 31 13:59:34.887809 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:34.887721    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 13:59:39.964686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:39.948115    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"]
Oct 31 13:59:39.964686 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:39.948408    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2" podUID="e425a959-4078-4fdc-88d2-8e6906eef5ab" containerName="maas-api" containerID="cri-o://a546919faa29bdb48aae76686113c5a2431de0b7db3708df9d115c20bce5e05f" gracePeriod=30
Oct 31 13:59:40.652962 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:40.652781    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:59:40.759648 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:40.759609    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e425a959-4078-4fdc-88d2-8e6906eef5ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e425a959-4078-4fdc-88d2-8e6906eef5ab-maas-api-tls\") pod \"e425a959-4078-4fdc-88d2-8e6906eef5ab\" (UID: \"e425a959-4078-4fdc-88d2-8e6906eef5ab\") "
Oct 31 13:59:40.759857 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:40.759821    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e425a959-4078-4fdc-88d2-8e6906eef5ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e425a959-4078-4fdc-88d2-8e6906eef5ab-openshift-service-ca\") pod \"e425a959-4078-4fdc-88d2-8e6906eef5ab\" (UID: \"e425a959-4078-4fdc-88d2-8e6906eef5ab\") "
Oct 31 13:59:40.759857 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:40.759850    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e425a959-4078-4fdc-88d2-8e6906eef5ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e425a959-4078-4fdc-88d2-8e6906eef5ab-metrics-certs\") pod \"e425a959-4078-4fdc-88d2-8e6906eef5ab\" (UID: \"e425a959-4078-4fdc-88d2-8e6906eef5ab\") "
Oct 31 13:59:40.760002 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:40.759904    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e425a959-4078-4fdc-88d2-8e6906eef5ab-kube-api-access-99r9d\" (UniqueName: \"kubernetes.io/projected/e425a959-4078-4fdc-88d2-8e6906eef5ab-kube-api-access-99r9d\") pod \"e425a959-4078-4fdc-88d2-8e6906eef5ab\" (UID: \"e425a959-4078-4fdc-88d2-8e6906eef5ab\") "
Oct 31 13:59:40.760213 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:40.760183    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e425a959-4078-4fdc-88d2-8e6906eef5ab-openshift-service-ca" pod "e425a959-4078-4fdc-88d2-8e6906eef5ab" (UID: "e425a959-4078-4fdc-88d2-8e6906eef5ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:59:40.760509 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:40.760480    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e425a959-4078-4fdc-88d2-8e6906eef5ab-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:40.762374 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:40.762344    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e425a959-4078-4fdc-88d2-8e6906eef5ab-kube-api-access-99r9d" pod "e425a959-4078-4fdc-88d2-8e6906eef5ab" (UID: "e425a959-4078-4fdc-88d2-8e6906eef5ab"). InnerVolumeSpecName "kube-api-access-99r9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:59:40.762604 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:40.762577    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e425a959-4078-4fdc-88d2-8e6906eef5ab-metrics-certs" pod "e425a959-4078-4fdc-88d2-8e6906eef5ab" (UID: "e425a959-4078-4fdc-88d2-8e6906eef5ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:59:40.762694 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:40.762602    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e425a959-4078-4fdc-88d2-8e6906eef5ab-maas-api-tls" pod "e425a959-4078-4fdc-88d2-8e6906eef5ab" (UID: "e425a959-4078-4fdc-88d2-8e6906eef5ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:59:40.870911 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:40.863681    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e425a959-4078-4fdc-88d2-8e6906eef5ab-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:40.870911 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:40.863720    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99r9d\" (UniqueName: \"kubernetes.io/projected/e425a959-4078-4fdc-88d2-8e6906eef5ab-kube-api-access-99r9d\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:40.870911 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:40.863737    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e425a959-4078-4fdc-88d2-8e6906eef5ab-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:41.073444 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:41.073411    4660 generic.go:356] "Generic (PLEG): container finished" podID="e425a959-4078-4fdc-88d2-8e6906eef5ab" containerID="a546919faa29bdb48aae76686113c5a2431de0b7db3708df9d115c20bce5e05f" exitCode=0
Oct 31 13:59:41.073857 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:41.073479    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"
Oct 31 13:59:41.073857 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:41.073499    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2" event={"ID":"e425a959-4078-4fdc-88d2-8e6906eef5ab","Type":"ContainerDied","Data":"a546919faa29bdb48aae76686113c5a2431de0b7db3708df9d115c20bce5e05f"}
Oct 31 13:59:41.073857 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:41.073593    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2" event={"ID":"e425a959-4078-4fdc-88d2-8e6906eef5ab","Type":"ContainerDied","Data":"7845f0ab8df0a4c0d8d4e0c6415e5900cca65d0bf258e2126e2443f855bd6547"}
Oct 31 13:59:41.073857 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:41.073621    4660 scope.go:122] "RemoveContainer" containerID="a546919faa29bdb48aae76686113c5a2431de0b7db3708df9d115c20bce5e05f"
Oct 31 13:59:41.087262 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:41.087237    4660 scope.go:122] "RemoveContainer" containerID="a546919faa29bdb48aae76686113c5a2431de0b7db3708df9d115c20bce5e05f"
Oct 31 13:59:41.087579 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:59:41.087557    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a546919faa29bdb48aae76686113c5a2431de0b7db3708df9d115c20bce5e05f\": container with ID starting with a546919faa29bdb48aae76686113c5a2431de0b7db3708df9d115c20bce5e05f not found: ID does not exist" containerID="a546919faa29bdb48aae76686113c5a2431de0b7db3708df9d115c20bce5e05f"
Oct 31 13:59:41.087656 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:41.087593    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a546919faa29bdb48aae76686113c5a2431de0b7db3708df9d115c20bce5e05f"} err="failed to get container status \"a546919faa29bdb48aae76686113c5a2431de0b7db3708df9d115c20bce5e05f\": rpc error: code = NotFound desc = could not find container \"a546919faa29bdb48aae76686113c5a2431de0b7db3708df9d115c20bce5e05f\": container with ID starting with a546919faa29bdb48aae76686113c5a2431de0b7db3708df9d115c20bce5e05f not found: ID does not exist"
Oct 31 13:59:41.104528 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:41.104497    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"]
Oct 31 13:59:41.116859 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:41.116833    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7b3ddd-58d6c5fb7f-66mm2"]
Oct 31 13:59:42.529196 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:42.529160    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e425a959-4078-4fdc-88d2-8e6906eef5ab" path="/var/lib/kubelet/pods/e425a959-4078-4fdc-88d2-8e6906eef5ab/volumes"
Oct 31 13:59:48.697049 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:48.697009    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"]
Oct 31 13:59:48.697868 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:48.697833    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp" podUID="c290e457-707d-4d9a-89e0-208ccb07111e" containerName="istio-proxy" containerID="cri-o://1d88cb997d2eef40bb20ce84623c6f5169edec43e34aa73588a701b633cf723f" gracePeriod=30
Oct 31 13:59:50.765976 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:59:50.765836    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 13:59:51.682229 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:51.682188    4660 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:59:51.682432 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:51.682259    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp" podUID="c290e457-707d-4d9a-89e0-208ccb07111e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:59:53.407168 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:53.406784    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"]
Oct 31 13:59:54.109759 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.109717    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:59:54.194314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.193118    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c290e457-707d-4d9a-89e0-208ccb07111e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c290e457-707d-4d9a-89e0-208ccb07111e-istiod-ca-cert\") pod \"c290e457-707d-4d9a-89e0-208ccb07111e\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") "
Oct 31 13:59:54.194314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.193184    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c290e457-707d-4d9a-89e0-208ccb07111e-istio-token\" (UniqueName: \"kubernetes.io/projected/c290e457-707d-4d9a-89e0-208ccb07111e-istio-token\") pod \"c290e457-707d-4d9a-89e0-208ccb07111e\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") "
Oct 31 13:59:54.194314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.193270    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c290e457-707d-4d9a-89e0-208ccb07111e-kube-api-access-sv296\" (UniqueName: \"kubernetes.io/projected/c290e457-707d-4d9a-89e0-208ccb07111e-kube-api-access-sv296\") pod \"c290e457-707d-4d9a-89e0-208ccb07111e\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") "
Oct 31 13:59:54.194314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.193324    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-credential-socket\") pod \"c290e457-707d-4d9a-89e0-208ccb07111e\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") "
Oct 31 13:59:54.194314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.193355    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-istio-envoy\") pod \"c290e457-707d-4d9a-89e0-208ccb07111e\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") "
Oct 31 13:59:54.194314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.193519    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-istio-data\") pod \"c290e457-707d-4d9a-89e0-208ccb07111e\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") "
Oct 31 13:59:54.194314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.193663    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-workload-certs\") pod \"c290e457-707d-4d9a-89e0-208ccb07111e\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") "
Oct 31 13:59:54.194314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.193695    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c290e457-707d-4d9a-89e0-208ccb07111e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c290e457-707d-4d9a-89e0-208ccb07111e-istio-podinfo\") pod \"c290e457-707d-4d9a-89e0-208ccb07111e\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") "
Oct 31 13:59:54.194314 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.193734    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-workload-socket\") pod \"c290e457-707d-4d9a-89e0-208ccb07111e\" (UID: \"c290e457-707d-4d9a-89e0-208ccb07111e\") "
Oct 31 13:59:54.235789 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.222505    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c290e457-707d-4d9a-89e0-208ccb07111e-istiod-ca-cert" pod "c290e457-707d-4d9a-89e0-208ccb07111e" (UID: "c290e457-707d-4d9a-89e0-208ccb07111e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:59:54.235789 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.234382    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-istio-data" pod "c290e457-707d-4d9a-89e0-208ccb07111e" (UID: "c290e457-707d-4d9a-89e0-208ccb07111e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:54.266267 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.245258    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-workload-socket" pod "c290e457-707d-4d9a-89e0-208ccb07111e" (UID: "c290e457-707d-4d9a-89e0-208ccb07111e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:54.266267 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.246466    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-workload-certs" pod "c290e457-707d-4d9a-89e0-208ccb07111e" (UID: "c290e457-707d-4d9a-89e0-208ccb07111e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:54.266267 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.247019    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-credential-socket" pod "c290e457-707d-4d9a-89e0-208ccb07111e" (UID: "c290e457-707d-4d9a-89e0-208ccb07111e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:54.270370 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.269483    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c290e457-707d-4d9a-89e0-208ccb07111e-istio-podinfo" pod "c290e457-707d-4d9a-89e0-208ccb07111e" (UID: "c290e457-707d-4d9a-89e0-208ccb07111e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:59:54.289035 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.270945    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c290e457-707d-4d9a-89e0-208ccb07111e-kube-api-access-sv296" pod "c290e457-707d-4d9a-89e0-208ccb07111e" (UID: "c290e457-707d-4d9a-89e0-208ccb07111e"). InnerVolumeSpecName "kube-api-access-sv296". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:59:54.319115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.289708    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c290e457-707d-4d9a-89e0-208ccb07111e-istio-token" pod "c290e457-707d-4d9a-89e0-208ccb07111e" (UID: "c290e457-707d-4d9a-89e0-208ccb07111e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:59:54.319115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.299491    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:54.319115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.299522    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:54.319115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.299539    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c290e457-707d-4d9a-89e0-208ccb07111e-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:54.319115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.299553    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:54.319115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.299566    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c290e457-707d-4d9a-89e0-208ccb07111e-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:54.319115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.299579    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c290e457-707d-4d9a-89e0-208ccb07111e-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:54.319115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.299593    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv296\" (UniqueName: \"kubernetes.io/projected/c290e457-707d-4d9a-89e0-208ccb07111e-kube-api-access-sv296\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:54.319115 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.299608    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:54.373121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.372279    4660 generic.go:356] "Generic (PLEG): container finished" podID="c290e457-707d-4d9a-89e0-208ccb07111e" containerID="1d88cb997d2eef40bb20ce84623c6f5169edec43e34aa73588a701b633cf723f" exitCode=0
Oct 31 13:59:54.373121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.372442    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp" event={"ID":"c290e457-707d-4d9a-89e0-208ccb07111e","Type":"ContainerDied","Data":"1d88cb997d2eef40bb20ce84623c6f5169edec43e34aa73588a701b633cf723f"}
Oct 31 13:59:54.373121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.372489    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp" event={"ID":"c290e457-707d-4d9a-89e0-208ccb07111e","Type":"ContainerDied","Data":"75642706d8896974ee9a75cc3f5d75c105e8b2d3730ee3c99ab4b4060b15e862"}
Oct 31 13:59:54.373121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.372518    4660 scope.go:122] "RemoveContainer" containerID="1d88cb997d2eef40bb20ce84623c6f5169edec43e34aa73588a701b633cf723f"
Oct 31 13:59:54.373121 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.372683    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"
Oct 31 13:59:54.377528 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.377496    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-istio-envoy" pod "c290e457-707d-4d9a-89e0-208ccb07111e" (UID: "c290e457-707d-4d9a-89e0-208ccb07111e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:54.397215 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.396843    4660 scope.go:122] "RemoveContainer" containerID="1d88cb997d2eef40bb20ce84623c6f5169edec43e34aa73588a701b633cf723f"
Oct 31 13:59:54.397618 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:59:54.397528    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d88cb997d2eef40bb20ce84623c6f5169edec43e34aa73588a701b633cf723f\": container with ID starting with 1d88cb997d2eef40bb20ce84623c6f5169edec43e34aa73588a701b633cf723f not found: ID does not exist" containerID="1d88cb997d2eef40bb20ce84623c6f5169edec43e34aa73588a701b633cf723f"
Oct 31 13:59:54.397618 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.397572    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d88cb997d2eef40bb20ce84623c6f5169edec43e34aa73588a701b633cf723f"} err="failed to get container status \"1d88cb997d2eef40bb20ce84623c6f5169edec43e34aa73588a701b633cf723f\": rpc error: code = NotFound desc = could not find container \"1d88cb997d2eef40bb20ce84623c6f5169edec43e34aa73588a701b633cf723f\": container with ID starting with 1d88cb997d2eef40bb20ce84623c6f5169edec43e34aa73588a701b633cf723f not found: ID does not exist"
Oct 31 13:59:54.401507 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.401464    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c290e457-707d-4d9a-89e0-208ccb07111e-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:54.696382 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.696288    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"]
Oct 31 13:59:54.703864 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:54.703831    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7wvkp"]
Oct 31 13:59:56.533065 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:56.533031    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c290e457-707d-4d9a-89e0-208ccb07111e" path="/var/lib/kubelet/pods/c290e457-707d-4d9a-89e0-208ccb07111e/volumes"
Oct 31 13:59:57.039588 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:57.039531    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"]
Oct 31 13:59:57.043457 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:57.043424    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c795281d-z4zjn"]
Oct 31 13:59:58.411672 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:58.411631    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv" podUID="4afbe3c3-aee3-4ed6-a37b-95e3c96c1792" containerName="payload-pre-processing" containerID="cri-o://e4049819458d7f69509125a3acc406cc1226a14907e4a79462a5012f6fdd4b86" gracePeriod=25
Oct 31 13:59:58.532999 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:58.532958    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd0752f-b6ee-4fa6-8084-574035cffcb3" path="/var/lib/kubelet/pods/bcd0752f-b6ee-4fa6-8084-574035cffcb3/volumes"
Oct 31 13:59:58.751742 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:58.751709    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"
Oct 31 13:59:58.857428 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:58.857397    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4afbe3c3-aee3-4ed6-a37b-95e3c96c1792-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afbe3c3-aee3-4ed6-a37b-95e3c96c1792-plugins-config-volume\") pod \"4afbe3c3-aee3-4ed6-a37b-95e3c96c1792\" (UID: \"4afbe3c3-aee3-4ed6-a37b-95e3c96c1792\") "
Oct 31 13:59:58.857611 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:58.857596    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4afbe3c3-aee3-4ed6-a37b-95e3c96c1792-kube-api-access-85gsl\" (UniqueName: \"kubernetes.io/projected/4afbe3c3-aee3-4ed6-a37b-95e3c96c1792-kube-api-access-85gsl\") pod \"4afbe3c3-aee3-4ed6-a37b-95e3c96c1792\" (UID: \"4afbe3c3-aee3-4ed6-a37b-95e3c96c1792\") "
Oct 31 13:59:58.857780 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:58.857757    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4afbe3c3-aee3-4ed6-a37b-95e3c96c1792-plugins-config-volume" pod "4afbe3c3-aee3-4ed6-a37b-95e3c96c1792" (UID: "4afbe3c3-aee3-4ed6-a37b-95e3c96c1792"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:59:58.858059 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:58.858045    4660 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afbe3c3-aee3-4ed6-a37b-95e3c96c1792-plugins-config-volume\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:58.859677 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:58.859652    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afbe3c3-aee3-4ed6-a37b-95e3c96c1792-kube-api-access-85gsl" pod "4afbe3c3-aee3-4ed6-a37b-95e3c96c1792" (UID: "4afbe3c3-aee3-4ed6-a37b-95e3c96c1792"). InnerVolumeSpecName "kube-api-access-85gsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:59:58.959282 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:58.959205    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85gsl\" (UniqueName: \"kubernetes.io/projected/4afbe3c3-aee3-4ed6-a37b-95e3c96c1792-kube-api-access-85gsl\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:59.124284 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.124232    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11119-e4927a"]
Oct 31 13:59:59.128987 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.128954    4660 cpu_manager.go:414] "containerMap: removing container" podUID="e425a959-4078-4fdc-88d2-8e6906eef5ab" containerName="maas-api"
Oct 31 13:59:59.129189 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.129175    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e425a959-4078-4fdc-88d2-8e6906eef5ab" containerName="maas-api"
Oct 31 13:59:59.129334 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.129322    4660 cpu_manager.go:414] "containerMap: removing container" podUID="4afbe3c3-aee3-4ed6-a37b-95e3c96c1792" containerName="payload-pre-processing"
Oct 31 13:59:59.129429 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.129418    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4afbe3c3-aee3-4ed6-a37b-95e3c96c1792" containerName="payload-pre-processing"
Oct 31 13:59:59.129504 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.129495    4660 cpu_manager.go:414] "containerMap: removing container" podUID="c290e457-707d-4d9a-89e0-208ccb07111e" containerName="istio-proxy"
Oct 31 13:59:59.129576 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.129566    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c290e457-707d-4d9a-89e0-208ccb07111e" containerName="istio-proxy"
Oct 31 13:59:59.129888 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.129867    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="4afbe3c3-aee3-4ed6-a37b-95e3c96c1792" containerName="payload-pre-processing"
Oct 31 13:59:59.130047 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.130024    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="c290e457-707d-4d9a-89e0-208ccb07111e" containerName="istio-proxy"
Oct 31 13:59:59.130196 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.130187    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="e425a959-4078-4fdc-88d2-8e6906eef5ab" containerName="maas-api"
Oct 31 13:59:59.145275 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.145249    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11119-e4927a"]
Oct 31 13:59:59.145614 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.145549    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11119-e4927a"
Oct 31 13:59:59.162414 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.162367    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q8gq\" (UniqueName: \"kubernetes.io/projected/e625d898-c36f-444d-9d3d-1145040aceda-kube-api-access-8q8gq\") pod \"test-curl-11119-e4927a\" (UID: \"e625d898-c36f-444d-9d3d-1145040aceda\") " pod="opendatahub/test-curl-11119-e4927a"
Oct 31 13:59:59.171824 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.171251    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-whw9x\""
Oct 31 13:59:59.263991 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.263901    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q8gq\" (UniqueName: \"kubernetes.io/projected/e625d898-c36f-444d-9d3d-1145040aceda-kube-api-access-8q8gq\") pod \"test-curl-11119-e4927a\" (UID: \"e625d898-c36f-444d-9d3d-1145040aceda\") " pod="opendatahub/test-curl-11119-e4927a"
Oct 31 13:59:59.279916 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.279887    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q8gq\" (UniqueName: \"kubernetes.io/projected/e625d898-c36f-444d-9d3d-1145040aceda-kube-api-access-8q8gq\") pod \"test-curl-11119-e4927a\" (UID: \"e625d898-c36f-444d-9d3d-1145040aceda\") " pod="opendatahub/test-curl-11119-e4927a"
Oct 31 13:59:59.476448 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.476409    4660 generic.go:356] "Generic (PLEG): container finished" podID="4afbe3c3-aee3-4ed6-a37b-95e3c96c1792" containerID="e4049819458d7f69509125a3acc406cc1226a14907e4a79462a5012f6fdd4b86" exitCode=0
Oct 31 13:59:59.476915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.476458    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"
Oct 31 13:59:59.476915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.476501    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv" event={"ID":"4afbe3c3-aee3-4ed6-a37b-95e3c96c1792","Type":"ContainerDied","Data":"e4049819458d7f69509125a3acc406cc1226a14907e4a79462a5012f6fdd4b86"}
Oct 31 13:59:59.476915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.476560    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv" event={"ID":"4afbe3c3-aee3-4ed6-a37b-95e3c96c1792","Type":"ContainerDied","Data":"1ccaf81d4f987d186e47af6b9f81168d30bf3a82d663ec13824f945d921a5619"}
Oct 31 13:59:59.476915 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.476585    4660 scope.go:122] "RemoveContainer" containerID="e4049819458d7f69509125a3acc406cc1226a14907e4a79462a5012f6fdd4b86"
Oct 31 13:59:59.484438 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.484415    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11119-e4927a"
Oct 31 13:59:59.488223 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.488205    4660 scope.go:122] "RemoveContainer" containerID="e4049819458d7f69509125a3acc406cc1226a14907e4a79462a5012f6fdd4b86"
Oct 31 13:59:59.488529 ip-10-0-137-77 kubenswrapper[4660]: E1031 13:59:59.488507    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4049819458d7f69509125a3acc406cc1226a14907e4a79462a5012f6fdd4b86\": container with ID starting with e4049819458d7f69509125a3acc406cc1226a14907e4a79462a5012f6fdd4b86 not found: ID does not exist" containerID="e4049819458d7f69509125a3acc406cc1226a14907e4a79462a5012f6fdd4b86"
Oct 31 13:59:59.488590 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.488541    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4049819458d7f69509125a3acc406cc1226a14907e4a79462a5012f6fdd4b86"} err="failed to get container status \"e4049819458d7f69509125a3acc406cc1226a14907e4a79462a5012f6fdd4b86\": rpc error: code = NotFound desc = could not find container \"e4049819458d7f69509125a3acc406cc1226a14907e4a79462a5012f6fdd4b86\": container with ID starting with e4049819458d7f69509125a3acc406cc1226a14907e4a79462a5012f6fdd4b86 not found: ID does not exist"
Oct 31 13:59:59.508419 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.508391    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"]
Oct 31 13:59:59.512942 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.512919    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-5b6b49c45f-l9nnv"]
Oct 31 13:59:59.770385 ip-10-0-137-77 kubenswrapper[4660]: W1031 13:59:59.770340    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode625d898_c36f_444d_9d3d_1145040aceda.slice/crio-55dd2f4b57924121b43ffc8958e953d01e6b9b3f271e9574e31ed78a27555f99 WatchSource:0}: Error finding container 55dd2f4b57924121b43ffc8958e953d01e6b9b3f271e9574e31ed78a27555f99: Status 404 returned error can't find the container with id 55dd2f4b57924121b43ffc8958e953d01e6b9b3f271e9574e31ed78a27555f99
Oct 31 13:59:59.770560 ip-10-0-137-77 kubenswrapper[4660]: I1031 13:59:59.770535    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11119-e4927a"]
Oct 31 14:00:00.209006 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.208972    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q"]
Oct 31 14:00:00.222078 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.219372    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h"]
Oct 31 14:00:00.222078 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.219485    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q"
Oct 31 14:00:00.239523 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.239494    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q"]
Oct 31 14:00:00.239758 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.239739    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h"]
Oct 31 14:00:00.239878 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.239855    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h"
Oct 31 14:00:00.261181 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.261148    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:00:00.278829 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.278789    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs6vz\" (UniqueName: \"kubernetes.io/projected/c687b3e0-c154-4c13-9f6e-0e05e7db3ac5-kube-api-access-gs6vz\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q\" (UID: \"c687b3e0-c154-4c13-9f6e-0e05e7db3ac5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q"
Oct 31 14:00:00.278999 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.278922    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5839b96b-4f18-4d9a-93df-42da4ba9e666-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h\" (UID: \"5839b96b-4f18-4d9a-93df-42da4ba9e666\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h"
Oct 31 14:00:00.279740 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.279149    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf7lg\" (UniqueName: \"kubernetes.io/projected/5839b96b-4f18-4d9a-93df-42da4ba9e666-kube-api-access-jf7lg\") pod \"maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h\" (UID: \"5839b96b-4f18-4d9a-93df-42da4ba9e666\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h"
Oct 31 14:00:00.279740 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.279224    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c687b3e0-c154-4c13-9f6e-0e05e7db3ac5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q\" (UID: \"c687b3e0-c154-4c13-9f6e-0e05e7db3ac5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q"
Oct 31 14:00:00.283369 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.283344    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b"]
Oct 31 14:00:00.304585 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.304557    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b"]
Oct 31 14:00:00.304791 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.304777    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj"]
Oct 31 14:00:00.305654 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.305627    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b"
Oct 31 14:00:00.326378 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.324091    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj"]
Oct 31 14:00:00.326378 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.324249    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj"
Oct 31 14:00:00.380018 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.379988    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf7lg\" (UniqueName: \"kubernetes.io/projected/5839b96b-4f18-4d9a-93df-42da4ba9e666-kube-api-access-jf7lg\") pod \"maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h\" (UID: \"5839b96b-4f18-4d9a-93df-42da4ba9e666\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h"
Oct 31 14:00:00.380198 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.380042    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c687b3e0-c154-4c13-9f6e-0e05e7db3ac5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q\" (UID: \"c687b3e0-c154-4c13-9f6e-0e05e7db3ac5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q"
Oct 31 14:00:00.380198 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.380079    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvdzf\" (UniqueName: \"kubernetes.io/projected/2f143333-68f6-413a-a882-71ce428ba0b1-kube-api-access-kvdzf\") pod \"maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b\" (UID: \"2f143333-68f6-413a-a882-71ce428ba0b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b"
Oct 31 14:00:00.380198 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.380109    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs6vz\" (UniqueName: \"kubernetes.io/projected/c687b3e0-c154-4c13-9f6e-0e05e7db3ac5-kube-api-access-gs6vz\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q\" (UID: \"c687b3e0-c154-4c13-9f6e-0e05e7db3ac5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q"
Oct 31 14:00:00.380364 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.380344    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5839b96b-4f18-4d9a-93df-42da4ba9e666-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h\" (UID: \"5839b96b-4f18-4d9a-93df-42da4ba9e666\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h"
Oct 31 14:00:00.380442 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.380425    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w2f5\" (UniqueName: \"kubernetes.io/projected/72fd0bc3-694c-44cc-8309-f59b16ab2561-kube-api-access-8w2f5\") pod \"maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj\" (UID: \"72fd0bc3-694c-44cc-8309-f59b16ab2561\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj"
Oct 31 14:00:00.380505 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.380489    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72fd0bc3-694c-44cc-8309-f59b16ab2561-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj\" (UID: \"72fd0bc3-694c-44cc-8309-f59b16ab2561\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj"
Oct 31 14:00:00.380612 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.380600    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f143333-68f6-413a-a882-71ce428ba0b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b\" (UID: \"2f143333-68f6-413a-a882-71ce428ba0b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b"
Oct 31 14:00:00.381022 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.380998    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c687b3e0-c154-4c13-9f6e-0e05e7db3ac5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q\" (UID: \"c687b3e0-c154-4c13-9f6e-0e05e7db3ac5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q"
Oct 31 14:00:00.381068 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.381019    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5839b96b-4f18-4d9a-93df-42da4ba9e666-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h\" (UID: \"5839b96b-4f18-4d9a-93df-42da4ba9e666\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h"
Oct 31 14:00:00.395915 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.395886    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf7lg\" (UniqueName: \"kubernetes.io/projected/5839b96b-4f18-4d9a-93df-42da4ba9e666-kube-api-access-jf7lg\") pod \"maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h\" (UID: \"5839b96b-4f18-4d9a-93df-42da4ba9e666\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h"
Oct 31 14:00:00.396339 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.396319    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs6vz\" (UniqueName: \"kubernetes.io/projected/c687b3e0-c154-4c13-9f6e-0e05e7db3ac5-kube-api-access-gs6vz\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q\" (UID: \"c687b3e0-c154-4c13-9f6e-0e05e7db3ac5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q"
Oct 31 14:00:00.481890 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.481798    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w2f5\" (UniqueName: \"kubernetes.io/projected/72fd0bc3-694c-44cc-8309-f59b16ab2561-kube-api-access-8w2f5\") pod \"maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj\" (UID: \"72fd0bc3-694c-44cc-8309-f59b16ab2561\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj"
Oct 31 14:00:00.481890 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.481871    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72fd0bc3-694c-44cc-8309-f59b16ab2561-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj\" (UID: \"72fd0bc3-694c-44cc-8309-f59b16ab2561\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj"
Oct 31 14:00:00.482454 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.481959    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f143333-68f6-413a-a882-71ce428ba0b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b\" (UID: \"2f143333-68f6-413a-a882-71ce428ba0b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b"
Oct 31 14:00:00.482454 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.482102    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvdzf\" (UniqueName: \"kubernetes.io/projected/2f143333-68f6-413a-a882-71ce428ba0b1-kube-api-access-kvdzf\") pod \"maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b\" (UID: \"2f143333-68f6-413a-a882-71ce428ba0b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b"
Oct 31 14:00:00.482691 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.482663    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72fd0bc3-694c-44cc-8309-f59b16ab2561-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj\" (UID: \"72fd0bc3-694c-44cc-8309-f59b16ab2561\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj"
Oct 31 14:00:00.482794 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.482772    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f143333-68f6-413a-a882-71ce428ba0b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b\" (UID: \"2f143333-68f6-413a-a882-71ce428ba0b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b"
Oct 31 14:00:00.487629 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.487599    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11119-e4927a" event={"ID":"e625d898-c36f-444d-9d3d-1145040aceda","Type":"ContainerStarted","Data":"8c6c8d9be3946edb6bd32711ecdf862aba92d75f4a49537711380cedcbd7f06b"}
Oct 31 14:00:00.487739 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.487639    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11119-e4927a" event={"ID":"e625d898-c36f-444d-9d3d-1145040aceda","Type":"ContainerStarted","Data":"55dd2f4b57924121b43ffc8958e953d01e6b9b3f271e9574e31ed78a27555f99"}
Oct 31 14:00:00.495054 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.495022    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w2f5\" (UniqueName: \"kubernetes.io/projected/72fd0bc3-694c-44cc-8309-f59b16ab2561-kube-api-access-8w2f5\") pod \"maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj\" (UID: \"72fd0bc3-694c-44cc-8309-f59b16ab2561\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj"
Oct 31 14:00:00.495228 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.495203    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvdzf\" (UniqueName: \"kubernetes.io/projected/2f143333-68f6-413a-a882-71ce428ba0b1-kube-api-access-kvdzf\") pod \"maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b\" (UID: \"2f143333-68f6-413a-a882-71ce428ba0b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b"
Oct 31 14:00:00.522086 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.519838    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11119-e4927a" podStartSLOduration=1.519819337 podStartE2EDuration="1.519819337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:00.515090954 +0000 UTC m=+3400.133750118" watchObservedRunningTime="2026-10-31 14:00:00.519819337 +0000 UTC m=+3400.138478503"
Oct 31 14:00:00.533801 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.533762    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4afbe3c3-aee3-4ed6-a37b-95e3c96c1792" path="/var/lib/kubelet/pods/4afbe3c3-aee3-4ed6-a37b-95e3c96c1792/volumes"
Oct 31 14:00:00.598928 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.598895    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q"
Oct 31 14:00:00.605368 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.605339    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h"
Oct 31 14:00:00.624887 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.624668    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b"
Oct 31 14:00:00.645612 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.645415    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj"
Oct 31 14:00:00.977055 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:00.977022    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj"]
Oct 31 14:00:00.981416 ip-10-0-137-77 kubenswrapper[4660]: W1031 14:00:00.981392    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72fd0bc3_694c_44cc_8309_f59b16ab2561.slice/crio-3d910cbefc121351847994ab3a724202412ebc30b4e70257807ce32df5ed5307 WatchSource:0}: Error finding container 3d910cbefc121351847994ab3a724202412ebc30b4e70257807ce32df5ed5307: Status 404 returned error can't find the container with id 3d910cbefc121351847994ab3a724202412ebc30b4e70257807ce32df5ed5307
Oct 31 14:00:01.134350 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.134262    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h"]
Oct 31 14:00:01.137003 ip-10-0-137-77 kubenswrapper[4660]: W1031 14:00:01.136392    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5839b96b_4f18_4d9a_93df_42da4ba9e666.slice/crio-cd03483ec1836f1f85c116b0e992799cdbf07f34de6c872680ecff2914ef50ef WatchSource:0}: Error finding container cd03483ec1836f1f85c116b0e992799cdbf07f34de6c872680ecff2914ef50ef: Status 404 returned error can't find the container with id cd03483ec1836f1f85c116b0e992799cdbf07f34de6c872680ecff2914ef50ef
Oct 31 14:00:01.142605 ip-10-0-137-77 kubenswrapper[4660]: W1031 14:00:01.142575    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc687b3e0_c154_4c13_9f6e_0e05e7db3ac5.slice/crio-f52745c612d0ca4c6da2b420837fc01ba0124df78f6601b13e81d0ddb89f177d WatchSource:0}: Error finding container f52745c612d0ca4c6da2b420837fc01ba0124df78f6601b13e81d0ddb89f177d: Status 404 returned error can't find the container with id f52745c612d0ca4c6da2b420837fc01ba0124df78f6601b13e81d0ddb89f177d
Oct 31 14:00:01.144190 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.144167    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q"]
Oct 31 14:00:01.156149 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.156122    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b"]
Oct 31 14:00:01.158731 ip-10-0-137-77 kubenswrapper[4660]: W1031 14:00:01.158704    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f143333_68f6_413a_a882_71ce428ba0b1.slice/crio-f7cd01c9e57e196f7a453acce7bc85361563caaaf05a57ef136ff253b3347622 WatchSource:0}: Error finding container f7cd01c9e57e196f7a453acce7bc85361563caaaf05a57ef136ff253b3347622: Status 404 returned error can't find the container with id f7cd01c9e57e196f7a453acce7bc85361563caaaf05a57ef136ff253b3347622
Oct 31 14:00:01.496281 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.496241    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h" event={"ID":"5839b96b-4f18-4d9a-93df-42da4ba9e666","Type":"ContainerStarted","Data":"cd03483ec1836f1f85c116b0e992799cdbf07f34de6c872680ecff2914ef50ef"}
Oct 31 14:00:01.498080 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.498046    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj" event={"ID":"72fd0bc3-694c-44cc-8309-f59b16ab2561","Type":"ContainerStarted","Data":"3d910cbefc121351847994ab3a724202412ebc30b4e70257807ce32df5ed5307"}
Oct 31 14:00:01.499558 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.499531    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q" event={"ID":"c687b3e0-c154-4c13-9f6e-0e05e7db3ac5","Type":"ContainerStarted","Data":"f52745c612d0ca4c6da2b420837fc01ba0124df78f6601b13e81d0ddb89f177d"}
Oct 31 14:00:01.500802 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.500780    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b" event={"ID":"2f143333-68f6-413a-a882-71ce428ba0b1","Type":"ContainerStarted","Data":"f7cd01c9e57e196f7a453acce7bc85361563caaaf05a57ef136ff253b3347622"}
Oct 31 14:00:01.563959 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.563881    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11119-e4927a"]
Oct 31 14:00:01.570028 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.569997    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11119-e4927a"]
Oct 31 14:00:01.718306 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.718255    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11119-94aa10"]
Oct 31 14:00:01.719381 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.719354    4660 cpu_manager.go:414] "containerMap: removing container" podUID="e625d898-c36f-444d-9d3d-1145040aceda" containerName="test-curl-11119-e4927a"
Oct 31 14:00:01.719509 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.719383    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e625d898-c36f-444d-9d3d-1145040aceda" containerName="test-curl-11119-e4927a"
Oct 31 14:00:01.719620 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.719601    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="e625d898-c36f-444d-9d3d-1145040aceda" containerName="test-curl-11119-e4927a"
Oct 31 14:00:01.723968 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.723943    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11119-94aa10"
Oct 31 14:00:01.735273 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.734157    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11119-94aa10"]
Oct 31 14:00:01.803755 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.803719    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmb29\" (UniqueName: \"kubernetes.io/projected/d12e744e-b164-46ef-b604-eef4d6ff980d-kube-api-access-cmb29\") pod \"test-curl-11119-94aa10\" (UID: \"d12e744e-b164-46ef-b604-eef4d6ff980d\") " pod="opendatahub/test-curl-11119-94aa10"
Oct 31 14:00:01.905726 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.905684    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmb29\" (UniqueName: \"kubernetes.io/projected/d12e744e-b164-46ef-b604-eef4d6ff980d-kube-api-access-cmb29\") pod \"test-curl-11119-94aa10\" (UID: \"d12e744e-b164-46ef-b604-eef4d6ff980d\") " pod="opendatahub/test-curl-11119-94aa10"
Oct 31 14:00:01.918786 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:01.918739    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmb29\" (UniqueName: \"kubernetes.io/projected/d12e744e-b164-46ef-b604-eef4d6ff980d-kube-api-access-cmb29\") pod \"test-curl-11119-94aa10\" (UID: \"d12e744e-b164-46ef-b604-eef4d6ff980d\") " pod="opendatahub/test-curl-11119-94aa10"
Oct 31 14:00:02.043115 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.043080    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11119-94aa10"
Oct 31 14:00:02.332432 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.332402    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11119-94aa10"]
Oct 31 14:00:02.337467 ip-10-0-137-77 kubenswrapper[4660]: W1031 14:00:02.337438    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd12e744e_b164_46ef_b604_eef4d6ff980d.slice/crio-fde412c3d3a60ac85502852804b7a9de3b4b73c94a9a247145c5c9edbcf7c573 WatchSource:0}: Error finding container fde412c3d3a60ac85502852804b7a9de3b4b73c94a9a247145c5c9edbcf7c573: Status 404 returned error can't find the container with id fde412c3d3a60ac85502852804b7a9de3b4b73c94a9a247145c5c9edbcf7c573
Oct 31 14:00:02.513231 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.513196    4660 generic.go:356] "Generic (PLEG): container finished" podID="72fd0bc3-694c-44cc-8309-f59b16ab2561" containerID="c4a98b0adaae2d4d71bc2c96e47f24e337d1d608ba6e7659b08431504d2bce3e" exitCode=60
Oct 31 14:00:02.513639 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.513287    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj" event={"ID":"72fd0bc3-694c-44cc-8309-f59b16ab2561","Type":"ContainerDied","Data":"c4a98b0adaae2d4d71bc2c96e47f24e337d1d608ba6e7659b08431504d2bce3e"}
Oct 31 14:00:02.513689 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.513658    4660 scope.go:122] "RemoveContainer" containerID="c4a98b0adaae2d4d71bc2c96e47f24e337d1d608ba6e7659b08431504d2bce3e"
Oct 31 14:00:02.516035 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.515283    4660 generic.go:356] "Generic (PLEG): container finished" podID="c687b3e0-c154-4c13-9f6e-0e05e7db3ac5" containerID="e2e2664c8157fd00236e40024c98eae625f17e642d380b56d2c8b5dba0645a87" exitCode=60
Oct 31 14:00:02.516035 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.515413    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q" event={"ID":"c687b3e0-c154-4c13-9f6e-0e05e7db3ac5","Type":"ContainerDied","Data":"e2e2664c8157fd00236e40024c98eae625f17e642d380b56d2c8b5dba0645a87"}
Oct 31 14:00:02.516035 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.516006    4660 scope.go:122] "RemoveContainer" containerID="e2e2664c8157fd00236e40024c98eae625f17e642d380b56d2c8b5dba0645a87"
Oct 31 14:00:02.517819 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.517762    4660 generic.go:356] "Generic (PLEG): container finished" podID="2f143333-68f6-413a-a882-71ce428ba0b1" containerID="d4029feee5794a71c9190fad6c4be70f96bba160341860194bd73ba2aa36fd9b" exitCode=60
Oct 31 14:00:02.517886 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.517832    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b" event={"ID":"2f143333-68f6-413a-a882-71ce428ba0b1","Type":"ContainerDied","Data":"d4029feee5794a71c9190fad6c4be70f96bba160341860194bd73ba2aa36fd9b"}
Oct 31 14:00:02.518207 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.518142    4660 scope.go:122] "RemoveContainer" containerID="d4029feee5794a71c9190fad6c4be70f96bba160341860194bd73ba2aa36fd9b"
Oct 31 14:00:02.548890 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.548855    4660 generic.go:356] "Generic (PLEG): container finished" podID="5839b96b-4f18-4d9a-93df-42da4ba9e666" containerID="c4d195e6c1e80b606c43e0067041784d00bd6b21a5133416fa7da49852f47247" exitCode=60
Oct 31 14:00:02.549182 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.549158    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11119-e4927a" podUID="e625d898-c36f-444d-9d3d-1145040aceda" containerName="test-curl-11119-e4927a" containerID="cri-o://8c6c8d9be3946edb6bd32711ecdf862aba92d75f4a49537711380cedcbd7f06b" gracePeriod=2
Oct 31 14:00:02.549643 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.549618    4660 scope.go:122] "RemoveContainer" containerID="c4d195e6c1e80b606c43e0067041784d00bd6b21a5133416fa7da49852f47247"
Oct 31 14:00:02.554946 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.554862    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11119-94aa10" event={"ID":"d12e744e-b164-46ef-b604-eef4d6ff980d","Type":"ContainerStarted","Data":"8fffd0575e011819f6f469a8fea5e7fd3e12f4be37f58b9a9db2164fdda4479e"}
Oct 31 14:00:02.554946 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.554903    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11119-94aa10" event={"ID":"d12e744e-b164-46ef-b604-eef4d6ff980d","Type":"ContainerStarted","Data":"fde412c3d3a60ac85502852804b7a9de3b4b73c94a9a247145c5c9edbcf7c573"}
Oct 31 14:00:02.555071 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:02.554948    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h" event={"ID":"5839b96b-4f18-4d9a-93df-42da4ba9e666","Type":"ContainerDied","Data":"c4d195e6c1e80b606c43e0067041784d00bd6b21a5133416fa7da49852f47247"}
Oct 31 14:00:03.332862 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.332797    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11119-94aa10" podStartSLOduration=2.332778946 podStartE2EDuration="2.332778946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:02.621249171 +0000 UTC m=+3402.239908336" watchObservedRunningTime="2026-10-31 14:00:03.332778946 +0000 UTC m=+3402.951438110"
Oct 31 14:00:03.338718 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.338679    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11119-94aa10"]
Oct 31 14:00:03.344810 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.344772    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11119-94aa10"]
Oct 31 14:00:03.560974 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.560937    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj" event={"ID":"72fd0bc3-694c-44cc-8309-f59b16ab2561","Type":"ContainerStarted","Data":"f776f33cdbab8ba141edd6abae0f5744db51ec30adf7cda59b17b38d90897bca"}
Oct 31 14:00:03.563014 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.562988    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q" event={"ID":"c687b3e0-c154-4c13-9f6e-0e05e7db3ac5","Type":"ContainerStarted","Data":"7e814293be99a91310625d72eda6e6d68878a3d01fdb0b7c82b8603433d1c950"}
Oct 31 14:00:03.565131 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.565103    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b" event={"ID":"2f143333-68f6-413a-a882-71ce428ba0b1","Type":"ContainerStarted","Data":"1ebe19e957df7df2818c6e8a5d1f106dfe78cb410bc95e6f96ba2ac1872b1232"}
Oct 31 14:00:03.567204 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.567175    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h" event={"ID":"5839b96b-4f18-4d9a-93df-42da4ba9e666","Type":"ContainerStarted","Data":"0079404c8d23c230028f1f26e50b77c92bde77762fcd07204e59b2fcba9698b9"}
Oct 31 14:00:03.579996 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.579952    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj" podStartSLOduration=3.579939857 podStartE2EDuration="3.579939857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:03.579018452 +0000 UTC m=+3403.197677608" watchObservedRunningTime="2026-10-31 14:00:03.579939857 +0000 UTC m=+3403.198599020"
Oct 31 14:00:03.600419 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.600374    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q" podStartSLOduration=3.600362067 podStartE2EDuration="3.600362067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:03.597976712 +0000 UTC m=+3403.216635877" watchObservedRunningTime="2026-10-31 14:00:03.600362067 +0000 UTC m=+3403.219021230"
Oct 31 14:00:03.621787 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.621717    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b" podStartSLOduration=3.621698995 podStartE2EDuration="3.621698995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:03.618259004 +0000 UTC m=+3403.236918170" watchObservedRunningTime="2026-10-31 14:00:03.621698995 +0000 UTC m=+3403.240358160"
Oct 31 14:00:03.645075 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.645022    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h" podStartSLOduration=3.645009138 podStartE2EDuration="3.645009138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:03.640768762 +0000 UTC m=+3403.259427928" watchObservedRunningTime="2026-10-31 14:00:03.645009138 +0000 UTC m=+3403.263668328"
Oct 31 14:00:03.868888 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.868859    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"]
Oct 31 14:00:03.869647 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.869624    4660 cpu_manager.go:414] "containerMap: removing container" podUID="d12e744e-b164-46ef-b604-eef4d6ff980d" containerName="test-curl-11119-94aa10"
Oct 31 14:00:03.869782 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.869649    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12e744e-b164-46ef-b604-eef4d6ff980d" containerName="test-curl-11119-94aa10"
Oct 31 14:00:03.869894 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.869876    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12e744e-b164-46ef-b604-eef4d6ff980d" containerName="test-curl-11119-94aa10"
Oct 31 14:00:03.873463 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.873442    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"
Oct 31 14:00:03.881504 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.881472    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:00:03.887186 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.887159    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"]
Oct 31 14:00:03.954258 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.954225    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-273ce012-8s4w4\" (UID: \"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"
Oct 31 14:00:03.954470 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.954339    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-273ce012-8s4w4\" (UID: \"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"
Oct 31 14:00:03.954470 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:03.954372    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6zvn\" (UniqueName: \"kubernetes.io/projected/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-kube-api-access-f6zvn\") pod \"maas-api-revoke-sub-273ce012-8s4w4\" (UID: \"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"
Oct 31 14:00:04.055468 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.055428    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6zvn\" (UniqueName: \"kubernetes.io/projected/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-kube-api-access-f6zvn\") pod \"maas-api-revoke-sub-273ce012-8s4w4\" (UID: \"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"
Oct 31 14:00:04.055662 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.055621    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-273ce012-8s4w4\" (UID: \"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"
Oct 31 14:00:04.055731 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.055675    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-273ce012-8s4w4\" (UID: \"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"
Oct 31 14:00:04.056285 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.056258    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-273ce012-8s4w4\" (UID: \"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"
Oct 31 14:00:04.058018 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.057996    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-273ce012-8s4w4\" (UID: \"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"
Oct 31 14:00:04.068969 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.068889    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6zvn\" (UniqueName: \"kubernetes.io/projected/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-kube-api-access-f6zvn\") pod \"maas-api-revoke-sub-273ce012-8s4w4\" (UID: \"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"
Oct 31 14:00:04.191887 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.191855    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"
Oct 31 14:00:04.480743 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.480716    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"]
Oct 31 14:00:04.482248 ip-10-0-137-77 kubenswrapper[4660]: W1031 14:00:04.482218    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd98cce8_cedf_4e5c_957d_7f9c79d7f6f0.slice/crio-1d55dafafc26bfe984f33211e695da6c54d812a5e984bbb6abb6a919c4891422 WatchSource:0}: Error finding container 1d55dafafc26bfe984f33211e695da6c54d812a5e984bbb6abb6a919c4891422: Status 404 returned error can't find the container with id 1d55dafafc26bfe984f33211e695da6c54d812a5e984bbb6abb6a919c4891422
Oct 31 14:00:04.577812 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.577772    4660 generic.go:356] "Generic (PLEG): container finished" podID="e625d898-c36f-444d-9d3d-1145040aceda" containerID="8c6c8d9be3946edb6bd32711ecdf862aba92d75f4a49537711380cedcbd7f06b" exitCode=137
Oct 31 14:00:04.590005 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.589976    4660 generic.go:356] "Generic (PLEG): container finished" podID="72fd0bc3-694c-44cc-8309-f59b16ab2561" containerID="f776f33cdbab8ba141edd6abae0f5744db51ec30adf7cda59b17b38d90897bca" exitCode=60
Oct 31 14:00:04.590139 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.590053    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj" event={"ID":"72fd0bc3-694c-44cc-8309-f59b16ab2561","Type":"ContainerDied","Data":"f776f33cdbab8ba141edd6abae0f5744db51ec30adf7cda59b17b38d90897bca"}
Oct 31 14:00:04.590139 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.590137    4660 scope.go:122] "RemoveContainer" containerID="c4a98b0adaae2d4d71bc2c96e47f24e337d1d608ba6e7659b08431504d2bce3e"
Oct 31 14:00:04.590536 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.590510    4660 scope.go:122] "RemoveContainer" containerID="f776f33cdbab8ba141edd6abae0f5744db51ec30adf7cda59b17b38d90897bca"
Oct 31 14:00:04.591174 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:04.591042    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj_odh-ai-gateway-infra(72fd0bc3-694c-44cc-8309-f59b16ab2561)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj" podUID="72fd0bc3-694c-44cc-8309-f59b16ab2561"
Oct 31 14:00:04.591849 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.591823    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4" event={"ID":"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0","Type":"ContainerStarted","Data":"1d55dafafc26bfe984f33211e695da6c54d812a5e984bbb6abb6a919c4891422"}
Oct 31 14:00:04.594585 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.594517    4660 generic.go:356] "Generic (PLEG): container finished" podID="c687b3e0-c154-4c13-9f6e-0e05e7db3ac5" containerID="7e814293be99a91310625d72eda6e6d68878a3d01fdb0b7c82b8603433d1c950" exitCode=60
Oct 31 14:00:04.594675 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.594617    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q" event={"ID":"c687b3e0-c154-4c13-9f6e-0e05e7db3ac5","Type":"ContainerDied","Data":"7e814293be99a91310625d72eda6e6d68878a3d01fdb0b7c82b8603433d1c950"}
Oct 31 14:00:04.594907 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.594883    4660 scope.go:122] "RemoveContainer" containerID="7e814293be99a91310625d72eda6e6d68878a3d01fdb0b7c82b8603433d1c950"
Oct 31 14:00:04.595177 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:04.595157    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q_odh-ai-gateway-infra(c687b3e0-c154-4c13-9f6e-0e05e7db3ac5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q" podUID="c687b3e0-c154-4c13-9f6e-0e05e7db3ac5"
Oct 31 14:00:04.597643 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.597617    4660 generic.go:356] "Generic (PLEG): container finished" podID="2f143333-68f6-413a-a882-71ce428ba0b1" containerID="1ebe19e957df7df2818c6e8a5d1f106dfe78cb410bc95e6f96ba2ac1872b1232" exitCode=60
Oct 31 14:00:04.597789 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.597773    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b" event={"ID":"2f143333-68f6-413a-a882-71ce428ba0b1","Type":"ContainerDied","Data":"1ebe19e957df7df2818c6e8a5d1f106dfe78cb410bc95e6f96ba2ac1872b1232"}
Oct 31 14:00:04.598121 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.598105    4660 scope.go:122] "RemoveContainer" containerID="1ebe19e957df7df2818c6e8a5d1f106dfe78cb410bc95e6f96ba2ac1872b1232"
Oct 31 14:00:04.598451 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:04.598414    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b_odh-ai-gateway-infra(2f143333-68f6-413a-a882-71ce428ba0b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b" podUID="2f143333-68f6-413a-a882-71ce428ba0b1"
Oct 31 14:00:04.620539 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.601412    4660 generic.go:356] "Generic (PLEG): container finished" podID="5839b96b-4f18-4d9a-93df-42da4ba9e666" containerID="0079404c8d23c230028f1f26e50b77c92bde77762fcd07204e59b2fcba9698b9" exitCode=60
Oct 31 14:00:04.620539 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.601493    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h" event={"ID":"5839b96b-4f18-4d9a-93df-42da4ba9e666","Type":"ContainerDied","Data":"0079404c8d23c230028f1f26e50b77c92bde77762fcd07204e59b2fcba9698b9"}
Oct 31 14:00:04.620539 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.601547    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11119-94aa10" podUID="d12e744e-b164-46ef-b604-eef4d6ff980d" containerName="test-curl-11119-94aa10" containerID="cri-o://8fffd0575e011819f6f469a8fea5e7fd3e12f4be37f58b9a9db2164fdda4479e" gracePeriod=2
Oct 31 14:00:04.620539 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.601707    4660 scope.go:122] "RemoveContainer" containerID="0079404c8d23c230028f1f26e50b77c92bde77762fcd07204e59b2fcba9698b9"
Oct 31 14:00:04.620539 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:04.601944    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h_odh-ai-gateway-infra(5839b96b-4f18-4d9a-93df-42da4ba9e666)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h" podUID="5839b96b-4f18-4d9a-93df-42da4ba9e666"
Oct 31 14:00:04.620539 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.608180    4660 scope.go:122] "RemoveContainer" containerID="e2e2664c8157fd00236e40024c98eae625f17e642d380b56d2c8b5dba0645a87"
Oct 31 14:00:04.634507 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.634486    4660 scope.go:122] "RemoveContainer" containerID="d4029feee5794a71c9190fad6c4be70f96bba160341860194bd73ba2aa36fd9b"
Oct 31 14:00:04.737833 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.732909    4660 scope.go:122] "RemoveContainer" containerID="c4d195e6c1e80b606c43e0067041784d00bd6b21a5133416fa7da49852f47247"
Oct 31 14:00:04.794341 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.794316    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11119-e4927a"
Oct 31 14:00:04.876731 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.876700    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e625d898-c36f-444d-9d3d-1145040aceda-kube-api-access-8q8gq\" (UniqueName: \"kubernetes.io/projected/e625d898-c36f-444d-9d3d-1145040aceda-kube-api-access-8q8gq\") pod \"e625d898-c36f-444d-9d3d-1145040aceda\" (UID: \"e625d898-c36f-444d-9d3d-1145040aceda\") "
Oct 31 14:00:04.879140 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.879110    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e625d898-c36f-444d-9d3d-1145040aceda-kube-api-access-8q8gq" pod "e625d898-c36f-444d-9d3d-1145040aceda" (UID: "e625d898-c36f-444d-9d3d-1145040aceda"). InnerVolumeSpecName "kube-api-access-8q8gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:04.978332 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:04.978288    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q8gq\" (UniqueName: \"kubernetes.io/projected/e625d898-c36f-444d-9d3d-1145040aceda-kube-api-access-8q8gq\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:05.625584 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:05.625212    4660 scope.go:122] "RemoveContainer" containerID="8c6c8d9be3946edb6bd32711ecdf862aba92d75f4a49537711380cedcbd7f06b"
Oct 31 14:00:05.625584 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:05.625261    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11119-e4927a"
Oct 31 14:00:05.631517 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:05.631497    4660 scope.go:122] "RemoveContainer" containerID="f776f33cdbab8ba141edd6abae0f5744db51ec30adf7cda59b17b38d90897bca"
Oct 31 14:00:05.632279 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:05.632242    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj_odh-ai-gateway-infra(72fd0bc3-694c-44cc-8309-f59b16ab2561)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj" podUID="72fd0bc3-694c-44cc-8309-f59b16ab2561"
Oct 31 14:00:05.633277 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:05.633250    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4" event={"ID":"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0","Type":"ContainerStarted","Data":"8e5ea0b4bba94006346ff6218b974439ea00d0b8a9cc25e8eb2c250f4bfaf50b"}
Oct 31 14:00:05.635766 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:05.635746    4660 scope.go:122] "RemoveContainer" containerID="7e814293be99a91310625d72eda6e6d68878a3d01fdb0b7c82b8603433d1c950"
Oct 31 14:00:05.636114 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:05.636080    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q_odh-ai-gateway-infra(c687b3e0-c154-4c13-9f6e-0e05e7db3ac5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q" podUID="c687b3e0-c154-4c13-9f6e-0e05e7db3ac5"
Oct 31 14:00:05.637644 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:05.637626    4660 scope.go:122] "RemoveContainer" containerID="1ebe19e957df7df2818c6e8a5d1f106dfe78cb410bc95e6f96ba2ac1872b1232"
Oct 31 14:00:05.637947 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:05.637922    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b_odh-ai-gateway-infra(2f143333-68f6-413a-a882-71ce428ba0b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b" podUID="2f143333-68f6-413a-a882-71ce428ba0b1"
Oct 31 14:00:05.639350 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:05.639333    4660 scope.go:122] "RemoveContainer" containerID="0079404c8d23c230028f1f26e50b77c92bde77762fcd07204e59b2fcba9698b9"
Oct 31 14:00:05.639529 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:05.639513    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h_odh-ai-gateway-infra(5839b96b-4f18-4d9a-93df-42da4ba9e666)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h" podUID="5839b96b-4f18-4d9a-93df-42da4ba9e666"
Oct 31 14:00:05.744015 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:05.743972    4660 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11119-e4927a\" is forbidden: User \"system:node:ip-10-0-137-77.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-77.us-east-2.compute.internal' and this object" podUID="e625d898-c36f-444d-9d3d-1145040aceda" pod="opendatahub/test-curl-11119-e4927a"
Oct 31 14:00:06.531286 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:06.531247    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e625d898-c36f-444d-9d3d-1145040aceda" path="/var/lib/kubelet/pods/e625d898-c36f-444d-9d3d-1145040aceda/volumes"
Oct 31 14:00:06.653174 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:06.653142    4660 generic.go:356] "Generic (PLEG): container finished" podID="dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0" containerID="8e5ea0b4bba94006346ff6218b974439ea00d0b8a9cc25e8eb2c250f4bfaf50b" exitCode=0
Oct 31 14:00:06.653564 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:06.653230    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4" event={"ID":"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0","Type":"ContainerDied","Data":"8e5ea0b4bba94006346ff6218b974439ea00d0b8a9cc25e8eb2c250f4bfaf50b"}
Oct 31 14:00:06.863233 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:06.863207    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11119-94aa10"
Oct 31 14:00:06.865267 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:06.865243    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"
Oct 31 14:00:07.025586 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.025551    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-maas-api-openshift-ca\") pod \"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0\" (UID: \"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0\") "
Oct 31 14:00:07.025803 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.025697    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-maas-api-xks-ca\") pod \"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0\" (UID: \"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0\") "
Oct 31 14:00:07.025803 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.025740    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-kube-api-access-f6zvn\" (UniqueName: \"kubernetes.io/projected/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-kube-api-access-f6zvn\") pod \"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0\" (UID: \"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0\") "
Oct 31 14:00:07.025921 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.025867    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12e744e-b164-46ef-b604-eef4d6ff980d-kube-api-access-cmb29\" (UniqueName: \"kubernetes.io/projected/d12e744e-b164-46ef-b604-eef4d6ff980d-kube-api-access-cmb29\") pod \"d12e744e-b164-46ef-b604-eef4d6ff980d\" (UID: \"d12e744e-b164-46ef-b604-eef4d6ff980d\") "
Oct 31 14:00:07.026011 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.025981    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-maas-api-openshift-ca" pod "dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0" (UID: "dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:07.026671 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.026642    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-maas-api-openshift-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:07.028149 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.028118    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-kube-api-access-f6zvn" pod "dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0" (UID: "dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0"). InnerVolumeSpecName "kube-api-access-f6zvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:07.028512 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.028489    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-maas-api-xks-ca" pod "dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0" (UID: "dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:00:07.029011 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.028627    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12e744e-b164-46ef-b604-eef4d6ff980d-kube-api-access-cmb29" pod "d12e744e-b164-46ef-b604-eef4d6ff980d" (UID: "d12e744e-b164-46ef-b604-eef4d6ff980d"). InnerVolumeSpecName "kube-api-access-cmb29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:07.128066 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.128029    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-maas-api-xks-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:07.128066 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.128069    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6zvn\" (UniqueName: \"kubernetes.io/projected/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0-kube-api-access-f6zvn\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:07.128066 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.128086    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmb29\" (UniqueName: \"kubernetes.io/projected/d12e744e-b164-46ef-b604-eef4d6ff980d-kube-api-access-cmb29\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:07.669662 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.669632    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"
Oct 31 14:00:07.670080 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.669676    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4" event={"ID":"dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0","Type":"ContainerDied","Data":"1d55dafafc26bfe984f33211e695da6c54d812a5e984bbb6abb6a919c4891422"}
Oct 31 14:00:07.670080 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.669730    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d55dafafc26bfe984f33211e695da6c54d812a5e984bbb6abb6a919c4891422"
Oct 31 14:00:07.670990 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.670963    4660 generic.go:356] "Generic (PLEG): container finished" podID="d12e744e-b164-46ef-b604-eef4d6ff980d" containerID="8fffd0575e011819f6f469a8fea5e7fd3e12f4be37f58b9a9db2164fdda4479e" exitCode=137
Oct 31 14:00:07.671129 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.671005    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11119-94aa10"
Oct 31 14:00:07.671129 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.671055    4660 scope.go:122] "RemoveContainer" containerID="8fffd0575e011819f6f469a8fea5e7fd3e12f4be37f58b9a9db2164fdda4479e"
Oct 31 14:00:07.682674 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.682657    4660 scope.go:122] "RemoveContainer" containerID="8fffd0575e011819f6f469a8fea5e7fd3e12f4be37f58b9a9db2164fdda4479e"
Oct 31 14:00:07.682936 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:07.682915    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fffd0575e011819f6f469a8fea5e7fd3e12f4be37f58b9a9db2164fdda4479e\": container with ID starting with 8fffd0575e011819f6f469a8fea5e7fd3e12f4be37f58b9a9db2164fdda4479e not found: ID does not exist" containerID="8fffd0575e011819f6f469a8fea5e7fd3e12f4be37f58b9a9db2164fdda4479e"
Oct 31 14:00:07.683004 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:07.682944    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fffd0575e011819f6f469a8fea5e7fd3e12f4be37f58b9a9db2164fdda4479e"} err="failed to get container status \"8fffd0575e011819f6f469a8fea5e7fd3e12f4be37f58b9a9db2164fdda4479e\": rpc error: code = NotFound desc = could not find container \"8fffd0575e011819f6f469a8fea5e7fd3e12f4be37f58b9a9db2164fdda4479e\": container with ID starting with 8fffd0575e011819f6f469a8fea5e7fd3e12f4be37f58b9a9db2164fdda4479e not found: ID does not exist"
Oct 31 14:00:07.689168 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:07.689143    4660 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11119-94aa10\" is forbidden: User \"system:node:ip-10-0-137-77.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-77.us-east-2.compute.internal' and this object" podUID="d12e744e-b164-46ef-b604-eef4d6ff980d" pod="opendatahub/test-curl-11119-94aa10"
Oct 31 14:00:07.691620 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:07.691597    4660 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11119-94aa10\" is forbidden: User \"system:node:ip-10-0-137-77.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-77.us-east-2.compute.internal' and this object" podUID="d12e744e-b164-46ef-b604-eef4d6ff980d" pod="opendatahub/test-curl-11119-94aa10"
Oct 31 14:00:08.561878 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:08.561839    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12e744e-b164-46ef-b604-eef4d6ff980d" path="/var/lib/kubelet/pods/d12e744e-b164-46ef-b604-eef4d6ff980d/volumes"
Oct 31 14:00:14.102528 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.102491    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"]
Oct 31 14:00:14.103168 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.103152    4660 cpu_manager.go:414] "containerMap: removing container" podUID="dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0" containerName="revoke-keys"
Oct 31 14:00:14.103219 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.103171    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0" containerName="revoke-keys"
Oct 31 14:00:14.103341 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.103330    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0" containerName="revoke-keys"
Oct 31 14:00:14.108419 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.108399    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"
Oct 31 14:00:14.121369 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.115725    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:00:14.121369 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.120518    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"]
Oct 31 14:00:14.166453 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.166415    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1281827e-5601-49c6-8822-a48e736b47ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c5f7a1-k2qvb\" (UID: \"1281827e-5601-49c6-8822-a48e736b47ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"
Oct 31 14:00:14.166695 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.166472    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1281827e-5601-49c6-8822-a48e736b47ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c5f7a1-k2qvb\" (UID: \"1281827e-5601-49c6-8822-a48e736b47ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"
Oct 31 14:00:14.166695 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.166504    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrhq7\" (UniqueName: \"kubernetes.io/projected/1281827e-5601-49c6-8822-a48e736b47ee-kube-api-access-xrhq7\") pod \"maas-api-revoke-sub-71c5f7a1-k2qvb\" (UID: \"1281827e-5601-49c6-8822-a48e736b47ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"
Oct 31 14:00:14.268474 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.268433    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1281827e-5601-49c6-8822-a48e736b47ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c5f7a1-k2qvb\" (UID: \"1281827e-5601-49c6-8822-a48e736b47ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"
Oct 31 14:00:14.268658 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.268483    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1281827e-5601-49c6-8822-a48e736b47ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c5f7a1-k2qvb\" (UID: \"1281827e-5601-49c6-8822-a48e736b47ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"
Oct 31 14:00:14.268658 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.268517    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrhq7\" (UniqueName: \"kubernetes.io/projected/1281827e-5601-49c6-8822-a48e736b47ee-kube-api-access-xrhq7\") pod \"maas-api-revoke-sub-71c5f7a1-k2qvb\" (UID: \"1281827e-5601-49c6-8822-a48e736b47ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"
Oct 31 14:00:14.269337 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.269288    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1281827e-5601-49c6-8822-a48e736b47ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c5f7a1-k2qvb\" (UID: \"1281827e-5601-49c6-8822-a48e736b47ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"
Oct 31 14:00:14.271050 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.271025    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1281827e-5601-49c6-8822-a48e736b47ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c5f7a1-k2qvb\" (UID: \"1281827e-5601-49c6-8822-a48e736b47ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"
Oct 31 14:00:14.287540 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.287512    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrhq7\" (UniqueName: \"kubernetes.io/projected/1281827e-5601-49c6-8822-a48e736b47ee-kube-api-access-xrhq7\") pod \"maas-api-revoke-sub-71c5f7a1-k2qvb\" (UID: \"1281827e-5601-49c6-8822-a48e736b47ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"
Oct 31 14:00:14.379320 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.377396    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"]
Oct 31 14:00:14.379320 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.377667    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578" podUID="f426a22f-701e-46ba-8a6b-99e13ceebfb5" containerName="maas-api" containerID="cri-o://f7d379f57f1af6977007e6ff1bd4d6e35cb061fb8c71998d0b39a912d12d2502" gracePeriod=30
Oct 31 14:00:14.441427 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.426218    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"
Oct 31 14:00:14.761159 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.761121    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"]
Oct 31 14:00:14.788778 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.788597    4660 generic.go:356] "Generic (PLEG): container finished" podID="f426a22f-701e-46ba-8a6b-99e13ceebfb5" containerID="f7d379f57f1af6977007e6ff1bd4d6e35cb061fb8c71998d0b39a912d12d2502" exitCode=0
Oct 31 14:00:14.788778 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.788669    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578" event={"ID":"f426a22f-701e-46ba-8a6b-99e13ceebfb5","Type":"ContainerDied","Data":"f7d379f57f1af6977007e6ff1bd4d6e35cb061fb8c71998d0b39a912d12d2502"}
Oct 31 14:00:14.894424 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.894024    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 14:00:14.979316 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.979205    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f426a22f-701e-46ba-8a6b-99e13ceebfb5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f426a22f-701e-46ba-8a6b-99e13ceebfb5-maas-api-tls\") pod \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\" (UID: \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\") "
Oct 31 14:00:14.979500 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.979399    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f426a22f-701e-46ba-8a6b-99e13ceebfb5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f426a22f-701e-46ba-8a6b-99e13ceebfb5-metrics-certs\") pod \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\" (UID: \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\") "
Oct 31 14:00:14.979500 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.979445    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f426a22f-701e-46ba-8a6b-99e13ceebfb5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f426a22f-701e-46ba-8a6b-99e13ceebfb5-openshift-service-ca\") pod \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\" (UID: \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\") "
Oct 31 14:00:14.979636 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.979621    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f426a22f-701e-46ba-8a6b-99e13ceebfb5-kube-api-access-n799z\" (UniqueName: \"kubernetes.io/projected/f426a22f-701e-46ba-8a6b-99e13ceebfb5-kube-api-access-n799z\") pod \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\" (UID: \"f426a22f-701e-46ba-8a6b-99e13ceebfb5\") "
Oct 31 14:00:14.981560 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:14.981511    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f426a22f-701e-46ba-8a6b-99e13ceebfb5-openshift-service-ca" pod "f426a22f-701e-46ba-8a6b-99e13ceebfb5" (UID: "f426a22f-701e-46ba-8a6b-99e13ceebfb5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:15.015358 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:15.012929    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f426a22f-701e-46ba-8a6b-99e13ceebfb5-kube-api-access-n799z" pod "f426a22f-701e-46ba-8a6b-99e13ceebfb5" (UID: "f426a22f-701e-46ba-8a6b-99e13ceebfb5"). InnerVolumeSpecName "kube-api-access-n799z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:15.015358 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:15.013469    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f426a22f-701e-46ba-8a6b-99e13ceebfb5-metrics-certs" pod "f426a22f-701e-46ba-8a6b-99e13ceebfb5" (UID: "f426a22f-701e-46ba-8a6b-99e13ceebfb5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:00:15.015358 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:15.013832    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f426a22f-701e-46ba-8a6b-99e13ceebfb5-maas-api-tls" pod "f426a22f-701e-46ba-8a6b-99e13ceebfb5" (UID: "f426a22f-701e-46ba-8a6b-99e13ceebfb5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:00:15.086317 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:15.085736    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f426a22f-701e-46ba-8a6b-99e13ceebfb5-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:15.086317 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:15.085790    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f426a22f-701e-46ba-8a6b-99e13ceebfb5-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:15.086317 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:15.085805    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n799z\" (UniqueName: \"kubernetes.io/projected/f426a22f-701e-46ba-8a6b-99e13ceebfb5-kube-api-access-n799z\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:15.086317 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:15.085819    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f426a22f-701e-46ba-8a6b-99e13ceebfb5-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:15.520166 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:15.520131    4660 scope.go:122] "RemoveContainer" containerID="1ebe19e957df7df2818c6e8a5d1f106dfe78cb410bc95e6f96ba2ac1872b1232"
Oct 31 14:00:15.799042 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:15.799007    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"
Oct 31 14:00:15.799218 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:15.799012    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578" event={"ID":"f426a22f-701e-46ba-8a6b-99e13ceebfb5","Type":"ContainerDied","Data":"f8e6d6c8462c1f512740569921e0a25a8896fc50c80d72a600440b91244b5cfc"}
Oct 31 14:00:15.799218 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:15.799181    4660 scope.go:122] "RemoveContainer" containerID="f7d379f57f1af6977007e6ff1bd4d6e35cb061fb8c71998d0b39a912d12d2502"
Oct 31 14:00:15.801129 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:15.801102    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb" event={"ID":"1281827e-5601-49c6-8822-a48e736b47ee","Type":"ContainerStarted","Data":"3b4a1e31aefef15b9af9606aece14391d9dd0dcc7137b173081618063795ebba"}
Oct 31 14:00:15.801256 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:15.801150    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb" event={"ID":"1281827e-5601-49c6-8822-a48e736b47ee","Type":"ContainerStarted","Data":"5edee8184026ddcec94b29bfd7439b383ffff5fea8e3c495b32654ca2feb35e1"}
Oct 31 14:00:15.829062 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:15.829015    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb" podStartSLOduration=1.829001743 podStartE2EDuration="1.829001743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:15.824408045 +0000 UTC m=+3415.443067203" watchObservedRunningTime="2026-10-31 14:00:15.829001743 +0000 UTC m=+3415.447660905"
Oct 31 14:00:15.843921 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:15.843896    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"]
Oct 31 14:00:15.850241 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:15.850219    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5af6eb-588dd5654-8s578"]
Oct 31 14:00:16.520381 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:16.520349    4660 scope.go:122] "RemoveContainer" containerID="f776f33cdbab8ba141edd6abae0f5744db51ec30adf7cda59b17b38d90897bca"
Oct 31 14:00:16.520738 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:16.520569    4660 scope.go:122] "RemoveContainer" containerID="7e814293be99a91310625d72eda6e6d68878a3d01fdb0b7c82b8603433d1c950"
Oct 31 14:00:16.531027 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:16.530995    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f426a22f-701e-46ba-8a6b-99e13ceebfb5" path="/var/lib/kubelet/pods/f426a22f-701e-46ba-8a6b-99e13ceebfb5/volumes"
Oct 31 14:00:16.811256 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:16.811219    4660 generic.go:356] "Generic (PLEG): container finished" podID="2f143333-68f6-413a-a882-71ce428ba0b1" containerID="cd13b70b418ea72453ff4044624ffa15cbb2c885dbf8634ec3758e0e7d07ebe1" exitCode=60
Oct 31 14:00:16.811429 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:16.811325    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b" event={"ID":"2f143333-68f6-413a-a882-71ce428ba0b1","Type":"ContainerDied","Data":"cd13b70b418ea72453ff4044624ffa15cbb2c885dbf8634ec3758e0e7d07ebe1"}
Oct 31 14:00:16.811429 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:16.811405    4660 scope.go:122] "RemoveContainer" containerID="1ebe19e957df7df2818c6e8a5d1f106dfe78cb410bc95e6f96ba2ac1872b1232"
Oct 31 14:00:16.811853 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:16.811826    4660 scope.go:122] "RemoveContainer" containerID="cd13b70b418ea72453ff4044624ffa15cbb2c885dbf8634ec3758e0e7d07ebe1"
Oct 31 14:00:16.812255 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:16.812225    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b_odh-ai-gateway-infra(2f143333-68f6-413a-a882-71ce428ba0b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b" podUID="2f143333-68f6-413a-a882-71ce428ba0b1"
Oct 31 14:00:16.814129 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:16.814105    4660 generic.go:356] "Generic (PLEG): container finished" podID="1281827e-5601-49c6-8822-a48e736b47ee" containerID="3b4a1e31aefef15b9af9606aece14391d9dd0dcc7137b173081618063795ebba" exitCode=0
Oct 31 14:00:16.814218 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:16.814183    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb" event={"ID":"1281827e-5601-49c6-8822-a48e736b47ee","Type":"ContainerDied","Data":"3b4a1e31aefef15b9af9606aece14391d9dd0dcc7137b173081618063795ebba"}
Oct 31 14:00:17.520023 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:17.519995    4660 scope.go:122] "RemoveContainer" containerID="0079404c8d23c230028f1f26e50b77c92bde77762fcd07204e59b2fcba9698b9"
Oct 31 14:00:17.827289 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:17.827260    4660 generic.go:356] "Generic (PLEG): container finished" podID="c687b3e0-c154-4c13-9f6e-0e05e7db3ac5" containerID="fbc63ca7db903f6cc1bb89db531a868c656d3ac38a45cdfa9f928c5bf5f89266" exitCode=60
Oct 31 14:00:17.827732 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:17.827354    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q" event={"ID":"c687b3e0-c154-4c13-9f6e-0e05e7db3ac5","Type":"ContainerDied","Data":"fbc63ca7db903f6cc1bb89db531a868c656d3ac38a45cdfa9f928c5bf5f89266"}
Oct 31 14:00:17.827732 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:17.827443    4660 scope.go:122] "RemoveContainer" containerID="7e814293be99a91310625d72eda6e6d68878a3d01fdb0b7c82b8603433d1c950"
Oct 31 14:00:17.827916 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:17.827896    4660 scope.go:122] "RemoveContainer" containerID="fbc63ca7db903f6cc1bb89db531a868c656d3ac38a45cdfa9f928c5bf5f89266"
Oct 31 14:00:17.828344 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:17.828285    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q_odh-ai-gateway-infra(c687b3e0-c154-4c13-9f6e-0e05e7db3ac5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q" podUID="c687b3e0-c154-4c13-9f6e-0e05e7db3ac5"
Oct 31 14:00:17.831509 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:17.831488    4660 generic.go:356] "Generic (PLEG): container finished" podID="72fd0bc3-694c-44cc-8309-f59b16ab2561" containerID="d9a50b43ba0f211afcbd6c5643b84b27a0029dd80b192c83870c9da4bbdd03ad" exitCode=60
Oct 31 14:00:17.831616 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:17.831563    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj" event={"ID":"72fd0bc3-694c-44cc-8309-f59b16ab2561","Type":"ContainerDied","Data":"d9a50b43ba0f211afcbd6c5643b84b27a0029dd80b192c83870c9da4bbdd03ad"}
Oct 31 14:00:17.832065 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:17.832048    4660 scope.go:122] "RemoveContainer" containerID="d9a50b43ba0f211afcbd6c5643b84b27a0029dd80b192c83870c9da4bbdd03ad"
Oct 31 14:00:17.832459 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:17.832436    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj_odh-ai-gateway-infra(72fd0bc3-694c-44cc-8309-f59b16ab2561)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj" podUID="72fd0bc3-694c-44cc-8309-f59b16ab2561"
Oct 31 14:00:17.845036 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:17.845004    4660 scope.go:122] "RemoveContainer" containerID="f776f33cdbab8ba141edd6abae0f5744db51ec30adf7cda59b17b38d90897bca"
Oct 31 14:00:17.846695 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:17.846674    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b"]
Oct 31 14:00:18.031699 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.031675    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"
Oct 31 14:00:18.095907 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.095834    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b"
Oct 31 14:00:18.106150 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.106117    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1281827e-5601-49c6-8822-a48e736b47ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1281827e-5601-49c6-8822-a48e736b47ee-maas-api-openshift-ca\") pod \"1281827e-5601-49c6-8822-a48e736b47ee\" (UID: \"1281827e-5601-49c6-8822-a48e736b47ee\") "
Oct 31 14:00:18.106323 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.106214    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1281827e-5601-49c6-8822-a48e736b47ee-kube-api-access-xrhq7\" (UniqueName: \"kubernetes.io/projected/1281827e-5601-49c6-8822-a48e736b47ee-kube-api-access-xrhq7\") pod \"1281827e-5601-49c6-8822-a48e736b47ee\" (UID: \"1281827e-5601-49c6-8822-a48e736b47ee\") "
Oct 31 14:00:18.106520 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.106490    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1281827e-5601-49c6-8822-a48e736b47ee-maas-api-openshift-ca" pod "1281827e-5601-49c6-8822-a48e736b47ee" (UID: "1281827e-5601-49c6-8822-a48e736b47ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:18.106520 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.106507    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1281827e-5601-49c6-8822-a48e736b47ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1281827e-5601-49c6-8822-a48e736b47ee-maas-api-xks-ca\") pod \"1281827e-5601-49c6-8822-a48e736b47ee\" (UID: \"1281827e-5601-49c6-8822-a48e736b47ee\") "
Oct 31 14:00:18.107147 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.107125    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1281827e-5601-49c6-8822-a48e736b47ee-maas-api-openshift-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:18.108517 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.108489    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1281827e-5601-49c6-8822-a48e736b47ee-kube-api-access-xrhq7" pod "1281827e-5601-49c6-8822-a48e736b47ee" (UID: "1281827e-5601-49c6-8822-a48e736b47ee"). InnerVolumeSpecName "kube-api-access-xrhq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:18.108990 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.108970    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1281827e-5601-49c6-8822-a48e736b47ee-maas-api-xks-ca" pod "1281827e-5601-49c6-8822-a48e736b47ee" (UID: "1281827e-5601-49c6-8822-a48e736b47ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:00:18.208077 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.208040    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f143333-68f6-413a-a882-71ce428ba0b1-kube-api-access-kvdzf\" (UniqueName: \"kubernetes.io/projected/2f143333-68f6-413a-a882-71ce428ba0b1-kube-api-access-kvdzf\") pod \"2f143333-68f6-413a-a882-71ce428ba0b1\" (UID: \"2f143333-68f6-413a-a882-71ce428ba0b1\") "
Oct 31 14:00:18.208253 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.208164    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f143333-68f6-413a-a882-71ce428ba0b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f143333-68f6-413a-a882-71ce428ba0b1-maas-api-service-ca\") pod \"2f143333-68f6-413a-a882-71ce428ba0b1\" (UID: \"2f143333-68f6-413a-a882-71ce428ba0b1\") "
Oct 31 14:00:18.208584 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.208558    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f143333-68f6-413a-a882-71ce428ba0b1-maas-api-service-ca" pod "2f143333-68f6-413a-a882-71ce428ba0b1" (UID: "2f143333-68f6-413a-a882-71ce428ba0b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:18.208648 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.208613    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1281827e-5601-49c6-8822-a48e736b47ee-maas-api-xks-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:18.208648 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.208629    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrhq7\" (UniqueName: \"kubernetes.io/projected/1281827e-5601-49c6-8822-a48e736b47ee-kube-api-access-xrhq7\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:18.210184 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.210161    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f143333-68f6-413a-a882-71ce428ba0b1-kube-api-access-kvdzf" pod "2f143333-68f6-413a-a882-71ce428ba0b1" (UID: "2f143333-68f6-413a-a882-71ce428ba0b1"). InnerVolumeSpecName "kube-api-access-kvdzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:18.309718 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.309677    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f143333-68f6-413a-a882-71ce428ba0b1-maas-api-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:18.309718 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.309712    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvdzf\" (UniqueName: \"kubernetes.io/projected/2f143333-68f6-413a-a882-71ce428ba0b1-kube-api-access-kvdzf\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:18.843925 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.843841    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b"
Oct 31 14:00:18.843925 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.843878    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b" event={"ID":"2f143333-68f6-413a-a882-71ce428ba0b1","Type":"ContainerDied","Data":"f7cd01c9e57e196f7a453acce7bc85361563caaaf05a57ef136ff253b3347622"}
Oct 31 14:00:18.844476 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.843933    4660 scope.go:122] "RemoveContainer" containerID="cd13b70b418ea72453ff4044624ffa15cbb2c885dbf8634ec3758e0e7d07ebe1"
Oct 31 14:00:18.846505 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.846422    4660 generic.go:356] "Generic (PLEG): container finished" podID="5839b96b-4f18-4d9a-93df-42da4ba9e666" containerID="420cdaa020d977d10e95ba27b69ab5d467251c8a395e4d8585af473674b816ed" exitCode=60
Oct 31 14:00:18.847120 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.846481    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h" event={"ID":"5839b96b-4f18-4d9a-93df-42da4ba9e666","Type":"ContainerDied","Data":"420cdaa020d977d10e95ba27b69ab5d467251c8a395e4d8585af473674b816ed"}
Oct 31 14:00:18.847636 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.847618    4660 scope.go:122] "RemoveContainer" containerID="420cdaa020d977d10e95ba27b69ab5d467251c8a395e4d8585af473674b816ed"
Oct 31 14:00:18.847960 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:18.847939    4660 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h_odh-ai-gateway-infra(5839b96b-4f18-4d9a-93df-42da4ba9e666)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h" podUID="5839b96b-4f18-4d9a-93df-42da4ba9e666"
Oct 31 14:00:18.849610 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.849592    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"
Oct 31 14:00:18.849703 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.849624    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb" event={"ID":"1281827e-5601-49c6-8822-a48e736b47ee","Type":"ContainerDied","Data":"5edee8184026ddcec94b29bfd7439b383ffff5fea8e3c495b32654ca2feb35e1"}
Oct 31 14:00:18.849703 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.849668    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5edee8184026ddcec94b29bfd7439b383ffff5fea8e3c495b32654ca2feb35e1"
Oct 31 14:00:18.857688 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.857670    4660 scope.go:122] "RemoveContainer" containerID="0079404c8d23c230028f1f26e50b77c92bde77762fcd07204e59b2fcba9698b9"
Oct 31 14:00:18.863592 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.863566    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q"]
Oct 31 14:00:18.880756 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.880681    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b"]
Oct 31 14:00:18.887097 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.887070    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj"]
Oct 31 14:00:18.891998 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:18.891419    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9f9103-29890920-8hq6b"]
Oct 31 14:00:19.125146 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.125115    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q"
Oct 31 14:00:19.129635 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.129609    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj"
Oct 31 14:00:19.229642 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.229608    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c687b3e0-c154-4c13-9f6e-0e05e7db3ac5-kube-api-access-gs6vz\" (UniqueName: \"kubernetes.io/projected/c687b3e0-c154-4c13-9f6e-0e05e7db3ac5-kube-api-access-gs6vz\") pod \"c687b3e0-c154-4c13-9f6e-0e05e7db3ac5\" (UID: \"c687b3e0-c154-4c13-9f6e-0e05e7db3ac5\") "
Oct 31 14:00:19.229851 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.229791    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72fd0bc3-694c-44cc-8309-f59b16ab2561-kube-api-access-8w2f5\" (UniqueName: \"kubernetes.io/projected/72fd0bc3-694c-44cc-8309-f59b16ab2561-kube-api-access-8w2f5\") pod \"72fd0bc3-694c-44cc-8309-f59b16ab2561\" (UID: \"72fd0bc3-694c-44cc-8309-f59b16ab2561\") "
Oct 31 14:00:19.230158 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.230128    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72fd0bc3-694c-44cc-8309-f59b16ab2561-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72fd0bc3-694c-44cc-8309-f59b16ab2561-maas-api-service-ca\") pod \"72fd0bc3-694c-44cc-8309-f59b16ab2561\" (UID: \"72fd0bc3-694c-44cc-8309-f59b16ab2561\") "
Oct 31 14:00:19.230672 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.230600    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c687b3e0-c154-4c13-9f6e-0e05e7db3ac5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c687b3e0-c154-4c13-9f6e-0e05e7db3ac5-maas-api-service-ca\") pod \"c687b3e0-c154-4c13-9f6e-0e05e7db3ac5\" (UID: \"c687b3e0-c154-4c13-9f6e-0e05e7db3ac5\") "
Oct 31 14:00:19.231353 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.230633    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72fd0bc3-694c-44cc-8309-f59b16ab2561-maas-api-service-ca" pod "72fd0bc3-694c-44cc-8309-f59b16ab2561" (UID: "72fd0bc3-694c-44cc-8309-f59b16ab2561"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:19.231638 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.231603    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c687b3e0-c154-4c13-9f6e-0e05e7db3ac5-maas-api-service-ca" pod "c687b3e0-c154-4c13-9f6e-0e05e7db3ac5" (UID: "c687b3e0-c154-4c13-9f6e-0e05e7db3ac5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:19.232010 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.231990    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72fd0bc3-694c-44cc-8309-f59b16ab2561-maas-api-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:19.232111 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.232015    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c687b3e0-c154-4c13-9f6e-0e05e7db3ac5-maas-api-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:19.232697 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.232669    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c687b3e0-c154-4c13-9f6e-0e05e7db3ac5-kube-api-access-gs6vz" pod "c687b3e0-c154-4c13-9f6e-0e05e7db3ac5" (UID: "c687b3e0-c154-4c13-9f6e-0e05e7db3ac5"). InnerVolumeSpecName "kube-api-access-gs6vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:19.234335 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.234312    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72fd0bc3-694c-44cc-8309-f59b16ab2561-kube-api-access-8w2f5" pod "72fd0bc3-694c-44cc-8309-f59b16ab2561" (UID: "72fd0bc3-694c-44cc-8309-f59b16ab2561"). InnerVolumeSpecName "kube-api-access-8w2f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:19.333546 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.333514    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs6vz\" (UniqueName: \"kubernetes.io/projected/c687b3e0-c154-4c13-9f6e-0e05e7db3ac5-kube-api-access-gs6vz\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:19.333546 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.333545    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w2f5\" (UniqueName: \"kubernetes.io/projected/72fd0bc3-694c-44cc-8309-f59b16ab2561-kube-api-access-8w2f5\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:19.860915 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.860827    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj"
Oct 31 14:00:19.860915 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.860860    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj" event={"ID":"72fd0bc3-694c-44cc-8309-f59b16ab2561","Type":"ContainerDied","Data":"3d910cbefc121351847994ab3a724202412ebc30b4e70257807ce32df5ed5307"}
Oct 31 14:00:19.861450 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.860955    4660 scope.go:122] "RemoveContainer" containerID="d9a50b43ba0f211afcbd6c5643b84b27a0029dd80b192c83870c9da4bbdd03ad"
Oct 31 14:00:19.862846 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.862792    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q" event={"ID":"c687b3e0-c154-4c13-9f6e-0e05e7db3ac5","Type":"ContainerDied","Data":"f52745c612d0ca4c6da2b420837fc01ba0124df78f6601b13e81d0ddb89f177d"}
Oct 31 14:00:19.862938 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.862906    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q"
Oct 31 14:00:19.884407 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.884385    4660 scope.go:122] "RemoveContainer" containerID="fbc63ca7db903f6cc1bb89db531a868c656d3ac38a45cdfa9f928c5bf5f89266"
Oct 31 14:00:19.902662 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.902636    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q"]
Oct 31 14:00:19.912214 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.912191    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890920-x6w8q"]
Oct 31 14:00:19.918103 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.918080    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h"]
Oct 31 14:00:19.935091 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.935062    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj"]
Oct 31 14:00:19.950776 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:19.950744    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-403dff-29890920-6rtmj"]
Oct 31 14:00:20.116325 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:20.116239    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h"
Oct 31 14:00:20.246597 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:20.246557    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5839b96b-4f18-4d9a-93df-42da4ba9e666-kube-api-access-jf7lg\" (UniqueName: \"kubernetes.io/projected/5839b96b-4f18-4d9a-93df-42da4ba9e666-kube-api-access-jf7lg\") pod \"5839b96b-4f18-4d9a-93df-42da4ba9e666\" (UID: \"5839b96b-4f18-4d9a-93df-42da4ba9e666\") "
Oct 31 14:00:20.246803 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:20.246667    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5839b96b-4f18-4d9a-93df-42da4ba9e666-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5839b96b-4f18-4d9a-93df-42da4ba9e666-maas-api-service-ca\") pod \"5839b96b-4f18-4d9a-93df-42da4ba9e666\" (UID: \"5839b96b-4f18-4d9a-93df-42da4ba9e666\") "
Oct 31 14:00:20.247039 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:20.247014    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5839b96b-4f18-4d9a-93df-42da4ba9e666-maas-api-service-ca" pod "5839b96b-4f18-4d9a-93df-42da4ba9e666" (UID: "5839b96b-4f18-4d9a-93df-42da4ba9e666"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:20.247309 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:20.247275    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5839b96b-4f18-4d9a-93df-42da4ba9e666-maas-api-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:20.248822 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:20.248792    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5839b96b-4f18-4d9a-93df-42da4ba9e666-kube-api-access-jf7lg" pod "5839b96b-4f18-4d9a-93df-42da4ba9e666" (UID: "5839b96b-4f18-4d9a-93df-42da4ba9e666"). InnerVolumeSpecName "kube-api-access-jf7lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:20.348261 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:20.348225    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf7lg\" (UniqueName: \"kubernetes.io/projected/5839b96b-4f18-4d9a-93df-42da4ba9e666-kube-api-access-jf7lg\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:20.541534 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:20.541501    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f143333-68f6-413a-a882-71ce428ba0b1" path="/var/lib/kubelet/pods/2f143333-68f6-413a-a882-71ce428ba0b1/volumes"
Oct 31 14:00:20.542021 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:20.542006    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72fd0bc3-694c-44cc-8309-f59b16ab2561" path="/var/lib/kubelet/pods/72fd0bc3-694c-44cc-8309-f59b16ab2561/volumes"
Oct 31 14:00:20.542477 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:20.542462    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c687b3e0-c154-4c13-9f6e-0e05e7db3ac5" path="/var/lib/kubelet/pods/c687b3e0-c154-4c13-9f6e-0e05e7db3ac5/volumes"
Oct 31 14:00:20.742532 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:20.742491    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:00:20.892359 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:20.892322    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h" event={"ID":"5839b96b-4f18-4d9a-93df-42da4ba9e666","Type":"ContainerDied","Data":"cd03483ec1836f1f85c116b0e992799cdbf07f34de6c872680ecff2914ef50ef"}
Oct 31 14:00:20.892839 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:20.892396    4660 scope.go:122] "RemoveContainer" containerID="420cdaa020d977d10e95ba27b69ab5d467251c8a395e4d8585af473674b816ed"
Oct 31 14:00:20.892839 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:20.892341    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h"
Oct 31 14:00:20.919521 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:20.919491    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h"]
Oct 31 14:00:20.925904 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:20.925874    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f9ca72-29890920-cjd7h"]
Oct 31 14:00:22.498257 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.498220    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"]
Oct 31 14:00:22.498906 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.498887    4660 cpu_manager.go:414] "containerMap: removing container" podUID="72fd0bc3-694c-44cc-8309-f59b16ab2561" containerName="cleanup"
Oct 31 14:00:22.498954 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.498908    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72fd0bc3-694c-44cc-8309-f59b16ab2561" containerName="cleanup"
Oct 31 14:00:22.498954 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.498921    4660 cpu_manager.go:414] "containerMap: removing container" podUID="f426a22f-701e-46ba-8a6b-99e13ceebfb5" containerName="maas-api"
Oct 31 14:00:22.498954 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.498927    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f426a22f-701e-46ba-8a6b-99e13ceebfb5" containerName="maas-api"
Oct 31 14:00:22.498954 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.498941    4660 cpu_manager.go:414] "containerMap: removing container" podUID="c687b3e0-c154-4c13-9f6e-0e05e7db3ac5" containerName="cleanup"
Oct 31 14:00:22.498954 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.498946    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c687b3e0-c154-4c13-9f6e-0e05e7db3ac5" containerName="cleanup"
Oct 31 14:00:22.498954 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.498956    4660 cpu_manager.go:414] "containerMap: removing container" podUID="c687b3e0-c154-4c13-9f6e-0e05e7db3ac5" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.498961    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c687b3e0-c154-4c13-9f6e-0e05e7db3ac5" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.498972    4660 cpu_manager.go:414] "containerMap: removing container" podUID="5839b96b-4f18-4d9a-93df-42da4ba9e666" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.498976    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5839b96b-4f18-4d9a-93df-42da4ba9e666" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.498982    4660 cpu_manager.go:414] "containerMap: removing container" podUID="2f143333-68f6-413a-a882-71ce428ba0b1" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.498986    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f143333-68f6-413a-a882-71ce428ba0b1" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.498992    4660 cpu_manager.go:414] "containerMap: removing container" podUID="2f143333-68f6-413a-a882-71ce428ba0b1" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.498996    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f143333-68f6-413a-a882-71ce428ba0b1" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499003    4660 cpu_manager.go:414] "containerMap: removing container" podUID="1281827e-5601-49c6-8822-a48e736b47ee" containerName="revoke-keys"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499007    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1281827e-5601-49c6-8822-a48e736b47ee" containerName="revoke-keys"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499014    4660 cpu_manager.go:414] "containerMap: removing container" podUID="5839b96b-4f18-4d9a-93df-42da4ba9e666" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499018    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5839b96b-4f18-4d9a-93df-42da4ba9e666" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499024    4660 cpu_manager.go:414] "containerMap: removing container" podUID="c687b3e0-c154-4c13-9f6e-0e05e7db3ac5" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499030    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c687b3e0-c154-4c13-9f6e-0e05e7db3ac5" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499038    4660 cpu_manager.go:414] "containerMap: removing container" podUID="72fd0bc3-694c-44cc-8309-f59b16ab2561" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499043    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72fd0bc3-694c-44cc-8309-f59b16ab2561" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499064    4660 cpu_manager.go:414] "containerMap: removing container" podUID="2f143333-68f6-413a-a882-71ce428ba0b1" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499068    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f143333-68f6-413a-a882-71ce428ba0b1" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499153    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="5839b96b-4f18-4d9a-93df-42da4ba9e666" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499161    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f143333-68f6-413a-a882-71ce428ba0b1" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499166    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f143333-68f6-413a-a882-71ce428ba0b1" containerName="cleanup"
Oct 31 14:00:22.499167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499174    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="1281827e-5601-49c6-8822-a48e736b47ee" containerName="revoke-keys"
Oct 31 14:00:22.499864 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499180    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="5839b96b-4f18-4d9a-93df-42da4ba9e666" containerName="cleanup"
Oct 31 14:00:22.499864 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499185    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="c687b3e0-c154-4c13-9f6e-0e05e7db3ac5" containerName="cleanup"
Oct 31 14:00:22.499864 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499190    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="72fd0bc3-694c-44cc-8309-f59b16ab2561" containerName="cleanup"
Oct 31 14:00:22.499864 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499195    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="5839b96b-4f18-4d9a-93df-42da4ba9e666" containerName="cleanup"
Oct 31 14:00:22.499864 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499204    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="72fd0bc3-694c-44cc-8309-f59b16ab2561" containerName="cleanup"
Oct 31 14:00:22.499864 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499210    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="f426a22f-701e-46ba-8a6b-99e13ceebfb5" containerName="maas-api"
Oct 31 14:00:22.499864 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499216    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="c687b3e0-c154-4c13-9f6e-0e05e7db3ac5" containerName="cleanup"
Oct 31 14:00:22.499864 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.499222    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="72fd0bc3-694c-44cc-8309-f59b16ab2561" containerName="cleanup"
Oct 31 14:00:22.502801 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.502779    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.512596 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.512571    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-a39e7e-openshift-default-dockercfg-x278n\""
Oct 31 14:00:22.517433 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.517413    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"]
Oct 31 14:00:22.557869 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.557831    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5839b96b-4f18-4d9a-93df-42da4ba9e666" path="/var/lib/kubelet/pods/5839b96b-4f18-4d9a-93df-42da4ba9e666/volumes"
Oct 31 14:00:22.579626 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.579592    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-workload-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.579767 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.579652    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-data\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.579823 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.579786    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-credential-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.579859 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.579827    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-envoy\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.579987 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.579963    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-workload-certs\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.580099 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.580010    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-podinfo\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.580099 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.580074    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwh6r\" (UniqueName: \"kubernetes.io/projected/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-kube-api-access-hwh6r\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.580180 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.580149    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istiod-ca-cert\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.580250 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.580233    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-token\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.681156 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.681119    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-workload-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.681376 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.681196    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-data\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.681376 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.681240    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-credential-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.681376 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.681255    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-envoy\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.681376 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.681321    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-workload-certs\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.681376 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.681346    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-podinfo\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.681729 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.681387    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwh6r\" (UniqueName: \"kubernetes.io/projected/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-kube-api-access-hwh6r\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.681729 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.681417    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istiod-ca-cert\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.681729 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.681503    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-token\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.681729 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.681584    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-workload-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.681729 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.681626    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-data\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.682051 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.681789    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-credential-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.682051 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.681842    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-workload-certs\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.682219 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.682200    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istiod-ca-cert\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.683622 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.683598    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-envoy\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.683900 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.683882    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-podinfo\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.694566 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.694540    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-token\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.694925 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.694901    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwh6r\" (UniqueName: \"kubernetes.io/projected/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-kube-api-access-hwh6r\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:22.828513 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:22.828418    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:23.143022 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:23.142165    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"]
Oct 31 14:00:23.147047 ip-10-0-137-77 kubenswrapper[4660]: W1031 14:00:23.147010    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fe7652_1fd9_470e_b8f9_a3cbcc061a8c.slice/crio-f5c4703fece358416b9c967b81030bd7451169e5d0ed7a0e72dc66671f1873a9 WatchSource:0}: Error finding container f5c4703fece358416b9c967b81030bd7451169e5d0ed7a0e72dc66671f1873a9: Status 404 returned error can't find the container with id f5c4703fece358416b9c967b81030bd7451169e5d0ed7a0e72dc66671f1873a9
Oct 31 14:00:23.151821 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:23.151783    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 14:00:23.152008 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:23.151985    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 14:00:23.152195 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:23.152176    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 14:00:23.928503 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:23.928460    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9" event={"ID":"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c","Type":"ContainerStarted","Data":"7c50b79f3f6caacc98982fe5bb7ab1c4adeea14c806aca1b0589f52ec2c4a0a7"}
Oct 31 14:00:23.928983 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:23.928529    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9" event={"ID":"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c","Type":"ContainerStarted","Data":"f5c4703fece358416b9c967b81030bd7451169e5d0ed7a0e72dc66671f1873a9"}
Oct 31 14:00:23.954190 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:23.954138    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9" podStartSLOduration=1.954122612 podStartE2EDuration="1.954122612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:23.952837882 +0000 UTC m=+3423.571497046" watchObservedRunningTime="2026-10-31 14:00:23.954122612 +0000 UTC m=+3423.572781833"
Oct 31 14:00:24.832659 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:24.828840    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:24.834127 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:24.834091    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 14:00:24.834264 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:24.834162    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9" podUID="44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 14:00:25.836398 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:25.832630    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 14:00:25.836398 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:25.832680    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9" podUID="44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 14:00:26.832780 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:26.832732    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 14:00:26.833000 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:26.832820    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9" podUID="44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 14:00:27.833319 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:27.832681    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 14:00:27.833319 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:27.832774    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9" podUID="44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 14:00:28.836322 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:28.830822    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 14:00:28.836322 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:28.830891    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9" podUID="44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 14:00:29.829271 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:29.829230    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 14:00:29.829473 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:29.829323    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9" podUID="44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 14:00:30.835411 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:30.835372    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:30.835914 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:30.835871    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:30.836795 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:30.836770    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:00:40.086766 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:40.086724    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"]
Oct 31 14:00:40.087193 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:40.086956    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47" podUID="1b061377-3c76-4eb8-aa22-04425a306fe6" containerName="istio-proxy" containerID="cri-o://5939af1968b7653c731e7fae17ca0019275996f6cf9f381c9745e358c261823d" gracePeriod=30
Oct 31 14:00:45.594189 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.594155    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 14:00:45.699885 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.699457    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-workload-certs\") pod \"1b061377-3c76-4eb8-aa22-04425a306fe6\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") "
Oct 31 14:00:45.700488 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.700379    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-workload-certs" pod "1b061377-3c76-4eb8-aa22-04425a306fe6" (UID: "1b061377-3c76-4eb8-aa22-04425a306fe6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:00:45.700616 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.700463    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-podinfo\") pod \"1b061377-3c76-4eb8-aa22-04425a306fe6\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") "
Oct 31 14:00:45.700616 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.700549    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-token\" (UniqueName: \"kubernetes.io/projected/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-token\") pod \"1b061377-3c76-4eb8-aa22-04425a306fe6\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") "
Oct 31 14:00:45.700616 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.700585    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-workload-socket\") pod \"1b061377-3c76-4eb8-aa22-04425a306fe6\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") "
Oct 31 14:00:45.700792 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.700630    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-credential-socket\") pod \"1b061377-3c76-4eb8-aa22-04425a306fe6\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") "
Oct 31 14:00:45.700792 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.700665    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-data\") pod \"1b061377-3c76-4eb8-aa22-04425a306fe6\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") "
Oct 31 14:00:45.700895 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.700824    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b061377-3c76-4eb8-aa22-04425a306fe6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b061377-3c76-4eb8-aa22-04425a306fe6-istiod-ca-cert\") pod \"1b061377-3c76-4eb8-aa22-04425a306fe6\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") "
Oct 31 14:00:45.701158 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.700950    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-envoy\") pod \"1b061377-3c76-4eb8-aa22-04425a306fe6\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") "
Oct 31 14:00:45.701158 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.700986    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b061377-3c76-4eb8-aa22-04425a306fe6-kube-api-access-wrwfb\" (UniqueName: \"kubernetes.io/projected/1b061377-3c76-4eb8-aa22-04425a306fe6-kube-api-access-wrwfb\") pod \"1b061377-3c76-4eb8-aa22-04425a306fe6\" (UID: \"1b061377-3c76-4eb8-aa22-04425a306fe6\") "
Oct 31 14:00:45.702158 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.701368    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-credential-socket" pod "1b061377-3c76-4eb8-aa22-04425a306fe6" (UID: "1b061377-3c76-4eb8-aa22-04425a306fe6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:00:45.702158 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.701807    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.702158 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.701834    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.703500 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.703005    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b061377-3c76-4eb8-aa22-04425a306fe6-istiod-ca-cert" pod "1b061377-3c76-4eb8-aa22-04425a306fe6" (UID: "1b061377-3c76-4eb8-aa22-04425a306fe6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:45.703500 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.703232    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-data" pod "1b061377-3c76-4eb8-aa22-04425a306fe6" (UID: "1b061377-3c76-4eb8-aa22-04425a306fe6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:00:45.704048 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.704025    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-workload-socket" pod "1b061377-3c76-4eb8-aa22-04425a306fe6" (UID: "1b061377-3c76-4eb8-aa22-04425a306fe6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:00:45.717158 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.717122    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-token" pod "1b061377-3c76-4eb8-aa22-04425a306fe6" (UID: "1b061377-3c76-4eb8-aa22-04425a306fe6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:45.717432 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.717402    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-podinfo" pod "1b061377-3c76-4eb8-aa22-04425a306fe6" (UID: "1b061377-3c76-4eb8-aa22-04425a306fe6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:00:45.725812 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.725768    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-envoy" pod "1b061377-3c76-4eb8-aa22-04425a306fe6" (UID: "1b061377-3c76-4eb8-aa22-04425a306fe6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:00:45.726356 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.726328    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b061377-3c76-4eb8-aa22-04425a306fe6-kube-api-access-wrwfb" pod "1b061377-3c76-4eb8-aa22-04425a306fe6" (UID: "1b061377-3c76-4eb8-aa22-04425a306fe6"). InnerVolumeSpecName "kube-api-access-wrwfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:45.803240 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.803195    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.803240 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.803239    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrwfb\" (UniqueName: \"kubernetes.io/projected/1b061377-3c76-4eb8-aa22-04425a306fe6-kube-api-access-wrwfb\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.803556 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.803256    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.803556 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.803269    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.803556 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.803282    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.803556 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.803313    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b061377-3c76-4eb8-aa22-04425a306fe6-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.803556 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:45.803326    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b061377-3c76-4eb8-aa22-04425a306fe6-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:46.335317 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:46.316728    4660 generic.go:356] "Generic (PLEG): container finished" podID="1b061377-3c76-4eb8-aa22-04425a306fe6" containerID="5939af1968b7653c731e7fae17ca0019275996f6cf9f381c9745e358c261823d" exitCode=0
Oct 31 14:00:46.335317 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:46.316834    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47" event={"ID":"1b061377-3c76-4eb8-aa22-04425a306fe6","Type":"ContainerDied","Data":"5939af1968b7653c731e7fae17ca0019275996f6cf9f381c9745e358c261823d"}
Oct 31 14:00:46.335317 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:46.316886    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47" event={"ID":"1b061377-3c76-4eb8-aa22-04425a306fe6","Type":"ContainerDied","Data":"ec32cd02b199bf4273851aa10160b7740f26d33c6269256f438dc2c402f5cd46"}
Oct 31 14:00:46.335317 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:46.316915    4660 scope.go:122] "RemoveContainer" containerID="5939af1968b7653c731e7fae17ca0019275996f6cf9f381c9745e358c261823d"
Oct 31 14:00:46.335317 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:46.317085    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"
Oct 31 14:00:46.423755 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:46.423717    4660 scope.go:122] "RemoveContainer" containerID="5939af1968b7653c731e7fae17ca0019275996f6cf9f381c9745e358c261823d"
Oct 31 14:00:46.431373 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:46.431227    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5939af1968b7653c731e7fae17ca0019275996f6cf9f381c9745e358c261823d\": container with ID starting with 5939af1968b7653c731e7fae17ca0019275996f6cf9f381c9745e358c261823d not found: ID does not exist" containerID="5939af1968b7653c731e7fae17ca0019275996f6cf9f381c9745e358c261823d"
Oct 31 14:00:46.431373 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:46.431281    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5939af1968b7653c731e7fae17ca0019275996f6cf9f381c9745e358c261823d"} err="failed to get container status \"5939af1968b7653c731e7fae17ca0019275996f6cf9f381c9745e358c261823d\": rpc error: code = NotFound desc = could not find container \"5939af1968b7653c731e7fae17ca0019275996f6cf9f381c9745e358c261823d\": container with ID starting with 5939af1968b7653c731e7fae17ca0019275996f6cf9f381c9745e358c261823d not found: ID does not exist"
Oct 31 14:00:46.437169 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:46.437137    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"]
Oct 31 14:00:46.446095 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:46.441480    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-k9v47"]
Oct 31 14:00:46.536600 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:46.536561    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b061377-3c76-4eb8-aa22-04425a306fe6" path="/var/lib/kubelet/pods/1b061377-3c76-4eb8-aa22-04425a306fe6/volumes"
Oct 31 14:00:50.738393 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:00:50.738354    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:00:55.038455 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:55.038403    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"]
Oct 31 14:00:55.039726 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:55.039672    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv" podUID="a94c907a-8bdd-4b73-86ee-3c132831df1f" containerName="istio-proxy" containerID="cri-o://225651aaf07d0fb1509bf60dadc15531f7ce891a134aa4486a90157200af6e40" gracePeriod=30
Oct 31 14:00:58.579696 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:58.579658    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:00:58.580242 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:00:58.579720    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv" podUID="a94c907a-8bdd-4b73-86ee-3c132831df1f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:01:00.492353 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.492326    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 14:01:00.501246 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.501212    4660 generic.go:356] "Generic (PLEG): container finished" podID="a94c907a-8bdd-4b73-86ee-3c132831df1f" containerID="225651aaf07d0fb1509bf60dadc15531f7ce891a134aa4486a90157200af6e40" exitCode=0
Oct 31 14:01:00.501458 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.501265    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"
Oct 31 14:01:00.504380 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.504342    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv" event={"ID":"a94c907a-8bdd-4b73-86ee-3c132831df1f","Type":"ContainerDied","Data":"225651aaf07d0fb1509bf60dadc15531f7ce891a134aa4486a90157200af6e40"}
Oct 31 14:01:00.504529 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.504487    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv" event={"ID":"a94c907a-8bdd-4b73-86ee-3c132831df1f","Type":"ContainerDied","Data":"0a7ef347daec1e156d100e035aa71238523687802cc3a054ccb91c85d8744226"}
Oct 31 14:01:00.504529 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.504524    4660 scope.go:122] "RemoveContainer" containerID="225651aaf07d0fb1509bf60dadc15531f7ce891a134aa4486a90157200af6e40"
Oct 31 14:01:00.536791 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.536735    4660 scope.go:122] "RemoveContainer" containerID="225651aaf07d0fb1509bf60dadc15531f7ce891a134aa4486a90157200af6e40"
Oct 31 14:01:00.538644 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:01:00.538607    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225651aaf07d0fb1509bf60dadc15531f7ce891a134aa4486a90157200af6e40\": container with ID starting with 225651aaf07d0fb1509bf60dadc15531f7ce891a134aa4486a90157200af6e40 not found: ID does not exist" containerID="225651aaf07d0fb1509bf60dadc15531f7ce891a134aa4486a90157200af6e40"
Oct 31 14:01:00.538839 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.538658    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225651aaf07d0fb1509bf60dadc15531f7ce891a134aa4486a90157200af6e40"} err="failed to get container status \"225651aaf07d0fb1509bf60dadc15531f7ce891a134aa4486a90157200af6e40\": rpc error: code = NotFound desc = could not find container \"225651aaf07d0fb1509bf60dadc15531f7ce891a134aa4486a90157200af6e40\": container with ID starting with 225651aaf07d0fb1509bf60dadc15531f7ce891a134aa4486a90157200af6e40 not found: ID does not exist"
Oct 31 14:01:00.549255 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.549219    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-data\") pod \"a94c907a-8bdd-4b73-86ee-3c132831df1f\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") "
Oct 31 14:01:00.549570 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.549547    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-envoy\") pod \"a94c907a-8bdd-4b73-86ee-3c132831df1f\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") "
Oct 31 14:01:00.549808 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.549787    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-podinfo\") pod \"a94c907a-8bdd-4b73-86ee-3c132831df1f\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") "
Oct 31 14:01:00.549961 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.549943    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-workload-socket\") pod \"a94c907a-8bdd-4b73-86ee-3c132831df1f\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") "
Oct 31 14:01:00.550091 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.550075    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-credential-socket\") pod \"a94c907a-8bdd-4b73-86ee-3c132831df1f\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") "
Oct 31 14:01:00.550323 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.550279    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a94c907a-8bdd-4b73-86ee-3c132831df1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a94c907a-8bdd-4b73-86ee-3c132831df1f-istiod-ca-cert\") pod \"a94c907a-8bdd-4b73-86ee-3c132831df1f\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") "
Oct 31 14:01:00.550480 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.550452    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-credential-socket" pod "a94c907a-8bdd-4b73-86ee-3c132831df1f" (UID: "a94c907a-8bdd-4b73-86ee-3c132831df1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:00.550616 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.550596    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a94c907a-8bdd-4b73-86ee-3c132831df1f-kube-api-access-fkhb7\" (UniqueName: \"kubernetes.io/projected/a94c907a-8bdd-4b73-86ee-3c132831df1f-kube-api-access-fkhb7\") pod \"a94c907a-8bdd-4b73-86ee-3c132831df1f\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") "
Oct 31 14:01:00.550790 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.550773    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-workload-certs\") pod \"a94c907a-8bdd-4b73-86ee-3c132831df1f\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") "
Oct 31 14:01:00.550970 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.550955    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-token\" (UniqueName: \"kubernetes.io/projected/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-token\") pod \"a94c907a-8bdd-4b73-86ee-3c132831df1f\" (UID: \"a94c907a-8bdd-4b73-86ee-3c132831df1f\") "
Oct 31 14:01:00.551771 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.551498    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-workload-socket" pod "a94c907a-8bdd-4b73-86ee-3c132831df1f" (UID: "a94c907a-8bdd-4b73-86ee-3c132831df1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:00.552612 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.552385    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-envoy" pod "a94c907a-8bdd-4b73-86ee-3c132831df1f" (UID: "a94c907a-8bdd-4b73-86ee-3c132831df1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:00.552612 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.552577    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-data" pod "a94c907a-8bdd-4b73-86ee-3c132831df1f" (UID: "a94c907a-8bdd-4b73-86ee-3c132831df1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:00.552911 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.552885    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94c907a-8bdd-4b73-86ee-3c132831df1f-istiod-ca-cert" pod "a94c907a-8bdd-4b73-86ee-3c132831df1f" (UID: "a94c907a-8bdd-4b73-86ee-3c132831df1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:00.553148 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.553123    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-workload-certs" pod "a94c907a-8bdd-4b73-86ee-3c132831df1f" (UID: "a94c907a-8bdd-4b73-86ee-3c132831df1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:00.553508 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.553482    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-podinfo" pod "a94c907a-8bdd-4b73-86ee-3c132831df1f" (UID: "a94c907a-8bdd-4b73-86ee-3c132831df1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:01:00.553704 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.553685    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.553839 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.553823    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.553950 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.553935    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.554043 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.554030    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.554130 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.554119    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a94c907a-8bdd-4b73-86ee-3c132831df1f-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.554217 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.554205    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a94c907a-8bdd-4b73-86ee-3c132831df1f-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.560804 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.560752    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94c907a-8bdd-4b73-86ee-3c132831df1f-kube-api-access-fkhb7" pod "a94c907a-8bdd-4b73-86ee-3c132831df1f" (UID: "a94c907a-8bdd-4b73-86ee-3c132831df1f"). InnerVolumeSpecName "kube-api-access-fkhb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:00.563659 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.563628    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-token" pod "a94c907a-8bdd-4b73-86ee-3c132831df1f" (UID: "a94c907a-8bdd-4b73-86ee-3c132831df1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:00.655757 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.655711    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.655757 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.655746    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkhb7\" (UniqueName: \"kubernetes.io/projected/a94c907a-8bdd-4b73-86ee-3c132831df1f-kube-api-access-fkhb7\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.655757 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.655765    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a94c907a-8bdd-4b73-86ee-3c132831df1f-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.834986 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.834930    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"]
Oct 31 14:01:00.841337 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:00.841283    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-27vnv"]
Oct 31 14:01:02.527941 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:02.527894    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a94c907a-8bdd-4b73-86ee-3c132831df1f" path="/var/lib/kubelet/pods/a94c907a-8bdd-4b73-86ee-3c132831df1f/volumes"
Oct 31 14:01:07.988968 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:07.988036    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"]
Oct 31 14:01:07.988968 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:07.988310    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk" podUID="f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" containerName="istio-proxy" containerID="cri-o://4327c67ab5b09bd35e1f12e27ad8352a828b51155b2fb447bba2cc062ca047c2" gracePeriod=30
Oct 31 14:01:07.996052 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:07.996017    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"]
Oct 31 14:01:07.997664 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:07.996239    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp" podUID="146bb004-162d-4777-92b9-08b3120df4d1" containerName="istio-proxy" containerID="cri-o://95cbdaf62050d358226fef728a8f495a5bc76f63669868fcf2b6b7d4b381639e" gracePeriod=30
Oct 31 14:01:08.006158 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:08.006100    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"]
Oct 31 14:01:08.006700 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:08.006664    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l" podUID="cc3f6cec-989a-4415-8328-389d1610947f" containerName="istio-proxy" containerID="cri-o://3d264a2ded8a53a24ea86bccc98aefe7d71d90aa5b291dedc2f65d09fbf6d727" gracePeriod=30
Oct 31 14:01:09.165624 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:09.165575    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:01:09.166097 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:09.165647    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk" podUID="f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:01:09.998278 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:09.998243    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:01:09.998502 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:09.998320    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l" podUID="cc3f6cec-989a-4415-8328-389d1610947f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:01:10.203312 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:10.202528    4660 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:01:10.203312 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:10.202591    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp" podUID="146bb004-162d-4777-92b9-08b3120df4d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:01:13.379822 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.379746    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 14:01:13.440365 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.440167    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"]
Oct 31 14:01:13.440516 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.440408    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc" podUID="344707b1-2bf5-4481-8522-afe6d996c26c" containerName="istio-proxy" containerID="cri-o://84460f572da86226085fdde211fd222a3ab7b0fb7d4c7d523533ef909b60cb9b" gracePeriod=30
Oct 31 14:01:13.464608 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.461788    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 14:01:13.464608 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.463735    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 14:01:13.469050 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.469025    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-workload-socket\") pod \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") "
Oct 31 14:01:13.469256 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.469236    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-istio-data\") pod \"146bb004-162d-4777-92b9-08b3120df4d1\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") "
Oct 31 14:01:13.469456 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.469439    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc3f6cec-989a-4415-8328-389d1610947f-kube-api-access-flm7n\" (UniqueName: \"kubernetes.io/projected/cc3f6cec-989a-4415-8328-389d1610947f-kube-api-access-flm7n\") pod \"cc3f6cec-989a-4415-8328-389d1610947f\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") "
Oct 31 14:01:13.469582 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.469554    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-istio-data" pod "146bb004-162d-4777-92b9-08b3120df4d1" (UID: "146bb004-162d-4777-92b9-08b3120df4d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.469702 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.469685    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-istio-envoy\") pod \"cc3f6cec-989a-4415-8328-389d1610947f\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") "
Oct 31 14:01:13.470754 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.470737    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-credential-socket\") pod \"cc3f6cec-989a-4415-8328-389d1610947f\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") "
Oct 31 14:01:13.470873 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.470862    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-workload-socket\") pod \"cc3f6cec-989a-4415-8328-389d1610947f\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") "
Oct 31 14:01:13.470945 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.470936    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istiod-ca-cert\") pod \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") "
Oct 31 14:01:13.471019 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.471010    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146bb004-162d-4777-92b9-08b3120df4d1-kube-api-access-6768n\" (UniqueName: \"kubernetes.io/projected/146bb004-162d-4777-92b9-08b3120df4d1-kube-api-access-6768n\") pod \"146bb004-162d-4777-92b9-08b3120df4d1\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") "
Oct 31 14:01:13.471090 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.471081    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-credential-socket\") pod \"146bb004-162d-4777-92b9-08b3120df4d1\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") "
Oct 31 14:01:13.471160 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.471152    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-data\") pod \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") "
Oct 31 14:01:13.471251 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.471241    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-podinfo\") pod \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") "
Oct 31 14:01:13.471758 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.471402    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-workload-certs\") pod \"cc3f6cec-989a-4415-8328-389d1610947f\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") "
Oct 31 14:01:13.471888 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.471874    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-istio-envoy\") pod \"146bb004-162d-4777-92b9-08b3120df4d1\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") "
Oct 31 14:01:13.471992 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.471971    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146bb004-162d-4777-92b9-08b3120df4d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146bb004-162d-4777-92b9-08b3120df4d1-istiod-ca-cert\") pod \"146bb004-162d-4777-92b9-08b3120df4d1\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") "
Oct 31 14:01:13.472068 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.472059    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-workload-certs\") pod \"146bb004-162d-4777-92b9-08b3120df4d1\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") "
Oct 31 14:01:13.472142 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.472134    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc3f6cec-989a-4415-8328-389d1610947f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc3f6cec-989a-4415-8328-389d1610947f-istiod-ca-cert\") pod \"cc3f6cec-989a-4415-8328-389d1610947f\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") "
Oct 31 14:01:13.472211 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.472202    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146bb004-162d-4777-92b9-08b3120df4d1-istio-token\" (UniqueName: \"kubernetes.io/projected/146bb004-162d-4777-92b9-08b3120df4d1-istio-token\") pod \"146bb004-162d-4777-92b9-08b3120df4d1\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") "
Oct 31 14:01:13.472322 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.472310    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc3f6cec-989a-4415-8328-389d1610947f-istio-token\" (UniqueName: \"kubernetes.io/projected/cc3f6cec-989a-4415-8328-389d1610947f-istio-token\") pod \"cc3f6cec-989a-4415-8328-389d1610947f\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") "
Oct 31 14:01:13.472416 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.472407    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/146bb004-162d-4777-92b9-08b3120df4d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146bb004-162d-4777-92b9-08b3120df4d1-istio-podinfo\") pod \"146bb004-162d-4777-92b9-08b3120df4d1\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") "
Oct 31 14:01:13.472505 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.472496    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-credential-socket\") pod \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") "
Oct 31 14:01:13.472608 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.472598    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-token\" (UniqueName: \"kubernetes.io/projected/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-token\") pod \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") "
Oct 31 14:01:13.472697 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.472686    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-workload-certs\") pod \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") "
Oct 31 14:01:13.472777 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.472768    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-kube-api-access-6nm59\" (UniqueName: \"kubernetes.io/projected/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-kube-api-access-6nm59\") pod \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") "
Oct 31 14:01:13.472917 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.472903    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-envoy\") pod \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\" (UID: \"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8\") "
Oct 31 14:01:13.473039 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.473023    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc3f6cec-989a-4415-8328-389d1610947f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc3f6cec-989a-4415-8328-389d1610947f-istio-podinfo\") pod \"cc3f6cec-989a-4415-8328-389d1610947f\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") "
Oct 31 14:01:13.473154 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.473140    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-istio-data\") pod \"cc3f6cec-989a-4415-8328-389d1610947f\" (UID: \"cc3f6cec-989a-4415-8328-389d1610947f\") "
Oct 31 14:01:13.473353 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.473335    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-workload-socket\") pod \"146bb004-162d-4777-92b9-08b3120df4d1\" (UID: \"146bb004-162d-4777-92b9-08b3120df4d1\") "
Oct 31 14:01:13.474267 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.474245    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.474407 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.470165    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-workload-socket" pod "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" (UID: "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.474514 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.472992    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc3f6cec-989a-4415-8328-389d1610947f-istiod-ca-cert" pod "cc3f6cec-989a-4415-8328-389d1610947f" (UID: "cc3f6cec-989a-4415-8328-389d1610947f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:13.474599 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.473357    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istiod-ca-cert" pod "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" (UID: "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:13.474675 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.473597    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-workload-certs" pod "cc3f6cec-989a-4415-8328-389d1610947f" (UID: "cc3f6cec-989a-4415-8328-389d1610947f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.474781 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.473831    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-credential-socket" pod "cc3f6cec-989a-4415-8328-389d1610947f" (UID: "cc3f6cec-989a-4415-8328-389d1610947f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.474846 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.474025    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-workload-socket" pod "cc3f6cec-989a-4415-8328-389d1610947f" (UID: "cc3f6cec-989a-4415-8328-389d1610947f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.480820 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.480755    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-envoy" pod "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" (UID: "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.480820 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.480772    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc3f6cec-989a-4415-8328-389d1610947f-istio-podinfo" pod "cc3f6cec-989a-4415-8328-389d1610947f" (UID: "cc3f6cec-989a-4415-8328-389d1610947f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:01:13.481213 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.481031    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-istio-data" pod "cc3f6cec-989a-4415-8328-389d1610947f" (UID: "cc3f6cec-989a-4415-8328-389d1610947f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.481542 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.481211    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-workload-socket" pod "146bb004-162d-4777-92b9-08b3120df4d1" (UID: "146bb004-162d-4777-92b9-08b3120df4d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.481542 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.481466    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-workload-certs" pod "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" (UID: "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.482521 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.481593    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-istio-envoy" pod "146bb004-162d-4777-92b9-08b3120df4d1" (UID: "146bb004-162d-4777-92b9-08b3120df4d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.482521 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.481694    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-istio-envoy" pod "cc3f6cec-989a-4415-8328-389d1610947f" (UID: "cc3f6cec-989a-4415-8328-389d1610947f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.482521 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.481964    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-token" pod "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" (UID: "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:13.482521 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.481988    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146bb004-162d-4777-92b9-08b3120df4d1-istiod-ca-cert" pod "146bb004-162d-4777-92b9-08b3120df4d1" (UID: "146bb004-162d-4777-92b9-08b3120df4d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:13.482521 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.482200    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-workload-certs" pod "146bb004-162d-4777-92b9-08b3120df4d1" (UID: "146bb004-162d-4777-92b9-08b3120df4d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.482895 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.482721    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-credential-socket" pod "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" (UID: "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.482956 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.482942    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-data" pod "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" (UID: "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.485156 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.485115    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-credential-socket" pod "146bb004-162d-4777-92b9-08b3120df4d1" (UID: "146bb004-162d-4777-92b9-08b3120df4d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.485278 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.485187    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-podinfo" pod "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" (UID: "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:01:13.485582 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.485440    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146bb004-162d-4777-92b9-08b3120df4d1-istio-token" pod "146bb004-162d-4777-92b9-08b3120df4d1" (UID: "146bb004-162d-4777-92b9-08b3120df4d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:13.488920 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.488891    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/146bb004-162d-4777-92b9-08b3120df4d1-istio-podinfo" pod "146bb004-162d-4777-92b9-08b3120df4d1" (UID: "146bb004-162d-4777-92b9-08b3120df4d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:01:13.489288 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.489262    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc3f6cec-989a-4415-8328-389d1610947f-kube-api-access-flm7n" pod "cc3f6cec-989a-4415-8328-389d1610947f" (UID: "cc3f6cec-989a-4415-8328-389d1610947f"). InnerVolumeSpecName "kube-api-access-flm7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:13.495667 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.495614    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc3f6cec-989a-4415-8328-389d1610947f-istio-token" pod "cc3f6cec-989a-4415-8328-389d1610947f" (UID: "cc3f6cec-989a-4415-8328-389d1610947f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:13.496007 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.495922    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146bb004-162d-4777-92b9-08b3120df4d1-kube-api-access-6768n" pod "146bb004-162d-4777-92b9-08b3120df4d1" (UID: "146bb004-162d-4777-92b9-08b3120df4d1"). InnerVolumeSpecName "kube-api-access-6768n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:13.500097 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.499973    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-kube-api-access-6nm59" pod "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" (UID: "f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8"). InnerVolumeSpecName "kube-api-access-6nm59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575480    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flm7n\" (UniqueName: \"kubernetes.io/projected/cc3f6cec-989a-4415-8328-389d1610947f-kube-api-access-flm7n\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575514    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575529    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575543    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575557    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575570    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6768n\" (UniqueName: \"kubernetes.io/projected/146bb004-162d-4777-92b9-08b3120df4d1-kube-api-access-6768n\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575586    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575600    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575614    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575628    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575643    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575658    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146bb004-162d-4777-92b9-08b3120df4d1-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575672    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575689    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc3f6cec-989a-4415-8328-389d1610947f-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575702    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146bb004-162d-4777-92b9-08b3120df4d1-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575715    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc3f6cec-989a-4415-8328-389d1610947f-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575730    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146bb004-162d-4777-92b9-08b3120df4d1-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575742    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575755    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575766    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575774    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nm59\" (UniqueName: \"kubernetes.io/projected/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-kube-api-access-6nm59\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575783    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575792    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc3f6cec-989a-4415-8328-389d1610947f-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575800    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc3f6cec-989a-4415-8328-389d1610947f-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575808    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146bb004-162d-4777-92b9-08b3120df4d1-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.580205 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.575816    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.640216 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.640181    4660 generic.go:356] "Generic (PLEG): container finished" podID="f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" containerID="4327c67ab5b09bd35e1f12e27ad8352a828b51155b2fb447bba2cc062ca047c2" exitCode=0
Oct 31 14:01:13.640439 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.640247    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"
Oct 31 14:01:13.640439 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.640267    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk" event={"ID":"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8","Type":"ContainerDied","Data":"4327c67ab5b09bd35e1f12e27ad8352a828b51155b2fb447bba2cc062ca047c2"}
Oct 31 14:01:13.640439 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.640363    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk" event={"ID":"f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8","Type":"ContainerDied","Data":"d15cfec7798e0f8d4214e81d3584aeb16d095e2af774ac6a0918bb55e14429eb"}
Oct 31 14:01:13.640439 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.640391    4660 scope.go:122] "RemoveContainer" containerID="4327c67ab5b09bd35e1f12e27ad8352a828b51155b2fb447bba2cc062ca047c2"
Oct 31 14:01:13.642003 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.641959    4660 generic.go:356] "Generic (PLEG): container finished" podID="cc3f6cec-989a-4415-8328-389d1610947f" containerID="3d264a2ded8a53a24ea86bccc98aefe7d71d90aa5b291dedc2f65d09fbf6d727" exitCode=0
Oct 31 14:01:13.642085 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.642013    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l" event={"ID":"cc3f6cec-989a-4415-8328-389d1610947f","Type":"ContainerDied","Data":"3d264a2ded8a53a24ea86bccc98aefe7d71d90aa5b291dedc2f65d09fbf6d727"}
Oct 31 14:01:13.642152 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.642070    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l" event={"ID":"cc3f6cec-989a-4415-8328-389d1610947f","Type":"ContainerDied","Data":"6290ed345c073306aac99086ba343fcf1f8db9cb17d5715c281a360d91c2e67e"}
Oct 31 14:01:13.642152 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.642029    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"
Oct 31 14:01:13.646218 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.643677    4660 generic.go:356] "Generic (PLEG): container finished" podID="146bb004-162d-4777-92b9-08b3120df4d1" containerID="95cbdaf62050d358226fef728a8f495a5bc76f63669868fcf2b6b7d4b381639e" exitCode=0
Oct 31 14:01:13.646218 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.643709    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp" event={"ID":"146bb004-162d-4777-92b9-08b3120df4d1","Type":"ContainerDied","Data":"95cbdaf62050d358226fef728a8f495a5bc76f63669868fcf2b6b7d4b381639e"}
Oct 31 14:01:13.646218 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.643747    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp" event={"ID":"146bb004-162d-4777-92b9-08b3120df4d1","Type":"ContainerDied","Data":"b4bfcef3f586c49a700feeb0a755b46b06decbc183eaf0fd9072625a274e332f"}
Oct 31 14:01:13.646218 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.643805    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"
Oct 31 14:01:13.655357 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.655338    4660 scope.go:122] "RemoveContainer" containerID="4327c67ab5b09bd35e1f12e27ad8352a828b51155b2fb447bba2cc062ca047c2"
Oct 31 14:01:13.655698 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:01:13.655674    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4327c67ab5b09bd35e1f12e27ad8352a828b51155b2fb447bba2cc062ca047c2\": container with ID starting with 4327c67ab5b09bd35e1f12e27ad8352a828b51155b2fb447bba2cc062ca047c2 not found: ID does not exist" containerID="4327c67ab5b09bd35e1f12e27ad8352a828b51155b2fb447bba2cc062ca047c2"
Oct 31 14:01:13.655805 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.655708    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4327c67ab5b09bd35e1f12e27ad8352a828b51155b2fb447bba2cc062ca047c2"} err="failed to get container status \"4327c67ab5b09bd35e1f12e27ad8352a828b51155b2fb447bba2cc062ca047c2\": rpc error: code = NotFound desc = could not find container \"4327c67ab5b09bd35e1f12e27ad8352a828b51155b2fb447bba2cc062ca047c2\": container with ID starting with 4327c67ab5b09bd35e1f12e27ad8352a828b51155b2fb447bba2cc062ca047c2 not found: ID does not exist"
Oct 31 14:01:13.655805 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.655731    4660 scope.go:122] "RemoveContainer" containerID="3d264a2ded8a53a24ea86bccc98aefe7d71d90aa5b291dedc2f65d09fbf6d727"
Oct 31 14:01:13.666871 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.666856    4660 scope.go:122] "RemoveContainer" containerID="3d264a2ded8a53a24ea86bccc98aefe7d71d90aa5b291dedc2f65d09fbf6d727"
Oct 31 14:01:13.667142 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:01:13.667118    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d264a2ded8a53a24ea86bccc98aefe7d71d90aa5b291dedc2f65d09fbf6d727\": container with ID starting with 3d264a2ded8a53a24ea86bccc98aefe7d71d90aa5b291dedc2f65d09fbf6d727 not found: ID does not exist" containerID="3d264a2ded8a53a24ea86bccc98aefe7d71d90aa5b291dedc2f65d09fbf6d727"
Oct 31 14:01:13.667227 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.667152    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d264a2ded8a53a24ea86bccc98aefe7d71d90aa5b291dedc2f65d09fbf6d727"} err="failed to get container status \"3d264a2ded8a53a24ea86bccc98aefe7d71d90aa5b291dedc2f65d09fbf6d727\": rpc error: code = NotFound desc = could not find container \"3d264a2ded8a53a24ea86bccc98aefe7d71d90aa5b291dedc2f65d09fbf6d727\": container with ID starting with 3d264a2ded8a53a24ea86bccc98aefe7d71d90aa5b291dedc2f65d09fbf6d727 not found: ID does not exist"
Oct 31 14:01:13.667227 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.667173    4660 scope.go:122] "RemoveContainer" containerID="95cbdaf62050d358226fef728a8f495a5bc76f63669868fcf2b6b7d4b381639e"
Oct 31 14:01:13.679001 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.678982    4660 scope.go:122] "RemoveContainer" containerID="95cbdaf62050d358226fef728a8f495a5bc76f63669868fcf2b6b7d4b381639e"
Oct 31 14:01:13.679259 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:01:13.679242    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95cbdaf62050d358226fef728a8f495a5bc76f63669868fcf2b6b7d4b381639e\": container with ID starting with 95cbdaf62050d358226fef728a8f495a5bc76f63669868fcf2b6b7d4b381639e not found: ID does not exist" containerID="95cbdaf62050d358226fef728a8f495a5bc76f63669868fcf2b6b7d4b381639e"
Oct 31 14:01:13.679339 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.679264    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95cbdaf62050d358226fef728a8f495a5bc76f63669868fcf2b6b7d4b381639e"} err="failed to get container status \"95cbdaf62050d358226fef728a8f495a5bc76f63669868fcf2b6b7d4b381639e\": rpc error: code = NotFound desc = could not find container \"95cbdaf62050d358226fef728a8f495a5bc76f63669868fcf2b6b7d4b381639e\": container with ID starting with 95cbdaf62050d358226fef728a8f495a5bc76f63669868fcf2b6b7d4b381639e not found: ID does not exist"
Oct 31 14:01:13.681000 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.680971    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"]
Oct 31 14:01:13.684807 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.684784    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-8blfk"]
Oct 31 14:01:13.697384 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.697364    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"]
Oct 31 14:01:13.708609 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.708586    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-9h6xp"]
Oct 31 14:01:13.720930 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.720907    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"]
Oct 31 14:01:13.727072 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:13.727051    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-7wj4l"]
Oct 31 14:01:14.531034 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:14.530995    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146bb004-162d-4777-92b9-08b3120df4d1" path="/var/lib/kubelet/pods/146bb004-162d-4777-92b9-08b3120df4d1/volumes"
Oct 31 14:01:14.531612 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:14.531594    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc3f6cec-989a-4415-8328-389d1610947f" path="/var/lib/kubelet/pods/cc3f6cec-989a-4415-8328-389d1610947f/volumes"
Oct 31 14:01:14.532053 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:14.532038    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" path="/var/lib/kubelet/pods/f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8/volumes"
Oct 31 14:01:16.066110 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:16.066071    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:01:16.066638 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:16.066132    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc" podUID="344707b1-2bf5-4481-8522-afe6d996c26c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:01:17.666510 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:17.665057    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"]
Oct 31 14:01:17.666510 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:17.665346    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn" podUID="f82f37eb-0856-4af8-857b-b855316ee6b0" containerName="maas-api" containerID="cri-o://00f5a52eab85ed6cb7e1f4b2b14b8d491d826d19965a4cf0edd1cbe05adb54f8" gracePeriod=30
Oct 31 14:01:18.259821 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.259795    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 14:01:18.370669 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.370633    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82f37eb-0856-4af8-857b-b855316ee6b0-kube-api-access-7ld8g\" (UniqueName: \"kubernetes.io/projected/f82f37eb-0856-4af8-857b-b855316ee6b0-kube-api-access-7ld8g\") pod \"f82f37eb-0856-4af8-857b-b855316ee6b0\" (UID: \"f82f37eb-0856-4af8-857b-b855316ee6b0\") "
Oct 31 14:01:18.370873 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.370743    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f82f37eb-0856-4af8-857b-b855316ee6b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f82f37eb-0856-4af8-857b-b855316ee6b0-maas-api-tls\") pod \"f82f37eb-0856-4af8-857b-b855316ee6b0\" (UID: \"f82f37eb-0856-4af8-857b-b855316ee6b0\") "
Oct 31 14:01:18.370873 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.370835    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f82f37eb-0856-4af8-857b-b855316ee6b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f82f37eb-0856-4af8-857b-b855316ee6b0-openshift-service-ca\") pod \"f82f37eb-0856-4af8-857b-b855316ee6b0\" (UID: \"f82f37eb-0856-4af8-857b-b855316ee6b0\") "
Oct 31 14:01:18.371038 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.371021    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f82f37eb-0856-4af8-857b-b855316ee6b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82f37eb-0856-4af8-857b-b855316ee6b0-metrics-certs\") pod \"f82f37eb-0856-4af8-857b-b855316ee6b0\" (UID: \"f82f37eb-0856-4af8-857b-b855316ee6b0\") "
Oct 31 14:01:18.371288 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.371262    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f82f37eb-0856-4af8-857b-b855316ee6b0-openshift-service-ca" pod "f82f37eb-0856-4af8-857b-b855316ee6b0" (UID: "f82f37eb-0856-4af8-857b-b855316ee6b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:18.371750 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.371727    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f82f37eb-0856-4af8-857b-b855316ee6b0-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:18.374697 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.374640    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f82f37eb-0856-4af8-857b-b855316ee6b0-maas-api-tls" pod "f82f37eb-0856-4af8-857b-b855316ee6b0" (UID: "f82f37eb-0856-4af8-857b-b855316ee6b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:18.374697 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.374652    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82f37eb-0856-4af8-857b-b855316ee6b0-kube-api-access-7ld8g" pod "f82f37eb-0856-4af8-857b-b855316ee6b0" (UID: "f82f37eb-0856-4af8-857b-b855316ee6b0"). InnerVolumeSpecName "kube-api-access-7ld8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:18.376067 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.375912    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f82f37eb-0856-4af8-857b-b855316ee6b0-metrics-certs" pod "f82f37eb-0856-4af8-857b-b855316ee6b0" (UID: "f82f37eb-0856-4af8-857b-b855316ee6b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:18.474560 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.474471    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f82f37eb-0856-4af8-857b-b855316ee6b0-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:18.474560 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.474510    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ld8g\" (UniqueName: \"kubernetes.io/projected/f82f37eb-0856-4af8-857b-b855316ee6b0-kube-api-access-7ld8g\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:18.474560 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.474542    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f82f37eb-0856-4af8-857b-b855316ee6b0-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:18.694398 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.694348    4660 generic.go:356] "Generic (PLEG): container finished" podID="344707b1-2bf5-4481-8522-afe6d996c26c" containerID="84460f572da86226085fdde211fd222a3ab7b0fb7d4c7d523533ef909b60cb9b" exitCode=0
Oct 31 14:01:18.694838 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.694431    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc" event={"ID":"344707b1-2bf5-4481-8522-afe6d996c26c","Type":"ContainerDied","Data":"84460f572da86226085fdde211fd222a3ab7b0fb7d4c7d523533ef909b60cb9b"}
Oct 31 14:01:18.696321 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.696284    4660 generic.go:356] "Generic (PLEG): container finished" podID="f82f37eb-0856-4af8-857b-b855316ee6b0" containerID="00f5a52eab85ed6cb7e1f4b2b14b8d491d826d19965a4cf0edd1cbe05adb54f8" exitCode=0
Oct 31 14:01:18.696460 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.696337    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn" event={"ID":"f82f37eb-0856-4af8-857b-b855316ee6b0","Type":"ContainerDied","Data":"00f5a52eab85ed6cb7e1f4b2b14b8d491d826d19965a4cf0edd1cbe05adb54f8"}
Oct 31 14:01:18.696460 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.696378    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn" event={"ID":"f82f37eb-0856-4af8-857b-b855316ee6b0","Type":"ContainerDied","Data":"850c86c321661c26f8967831c5a6c3bc1ecc787a8d715ae119bf87239604b009"}
Oct 31 14:01:18.696460 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.696407    4660 scope.go:122] "RemoveContainer" containerID="00f5a52eab85ed6cb7e1f4b2b14b8d491d826d19965a4cf0edd1cbe05adb54f8"
Oct 31 14:01:18.698059 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.697953    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"
Oct 31 14:01:18.844369 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.844268    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"]
Oct 31 14:01:18.846015 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.845449    4660 scope.go:122] "RemoveContainer" containerID="00f5a52eab85ed6cb7e1f4b2b14b8d491d826d19965a4cf0edd1cbe05adb54f8"
Oct 31 14:01:18.858255 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.858216    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9ca72-6994d87787-t4rfn"]
Oct 31 14:01:18.867468 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:01:18.866779    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f5a52eab85ed6cb7e1f4b2b14b8d491d826d19965a4cf0edd1cbe05adb54f8\": container with ID starting with 00f5a52eab85ed6cb7e1f4b2b14b8d491d826d19965a4cf0edd1cbe05adb54f8 not found: ID does not exist" containerID="00f5a52eab85ed6cb7e1f4b2b14b8d491d826d19965a4cf0edd1cbe05adb54f8"
Oct 31 14:01:18.867468 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.866822    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f5a52eab85ed6cb7e1f4b2b14b8d491d826d19965a4cf0edd1cbe05adb54f8"} err="failed to get container status \"00f5a52eab85ed6cb7e1f4b2b14b8d491d826d19965a4cf0edd1cbe05adb54f8\": rpc error: code = NotFound desc = could not find container \"00f5a52eab85ed6cb7e1f4b2b14b8d491d826d19965a4cf0edd1cbe05adb54f8\": container with ID starting with 00f5a52eab85ed6cb7e1f4b2b14b8d491d826d19965a4cf0edd1cbe05adb54f8 not found: ID does not exist"
Oct 31 14:01:18.891740 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.891713    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 14:01:18.988086 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.988041    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/344707b1-2bf5-4481-8522-afe6d996c26c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344707b1-2bf5-4481-8522-afe6d996c26c-istiod-ca-cert\") pod \"344707b1-2bf5-4481-8522-afe6d996c26c\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") "
Oct 31 14:01:18.988283 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.988145    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-istio-envoy\") pod \"344707b1-2bf5-4481-8522-afe6d996c26c\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") "
Oct 31 14:01:18.988283 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.988194    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-workload-certs\") pod \"344707b1-2bf5-4481-8522-afe6d996c26c\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") "
Oct 31 14:01:18.988283 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.988245    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-workload-socket\") pod \"344707b1-2bf5-4481-8522-afe6d996c26c\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") "
Oct 31 14:01:18.988283 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.988272    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/344707b1-2bf5-4481-8522-afe6d996c26c-kube-api-access-hhxmf\" (UniqueName: \"kubernetes.io/projected/344707b1-2bf5-4481-8522-afe6d996c26c-kube-api-access-hhxmf\") pod \"344707b1-2bf5-4481-8522-afe6d996c26c\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") "
Oct 31 14:01:18.988585 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.988333    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/344707b1-2bf5-4481-8522-afe6d996c26c-istio-token\" (UniqueName: \"kubernetes.io/projected/344707b1-2bf5-4481-8522-afe6d996c26c-istio-token\") pod \"344707b1-2bf5-4481-8522-afe6d996c26c\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") "
Oct 31 14:01:18.988585 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.988385    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-istio-data\") pod \"344707b1-2bf5-4481-8522-afe6d996c26c\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") "
Oct 31 14:01:18.988585 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.988536    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-credential-socket\") pod \"344707b1-2bf5-4481-8522-afe6d996c26c\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") "
Oct 31 14:01:18.988585 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.988564    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/344707b1-2bf5-4481-8522-afe6d996c26c-istiod-ca-cert" pod "344707b1-2bf5-4481-8522-afe6d996c26c" (UID: "344707b1-2bf5-4481-8522-afe6d996c26c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:18.988793 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.988597    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/344707b1-2bf5-4481-8522-afe6d996c26c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344707b1-2bf5-4481-8522-afe6d996c26c-istio-podinfo\") pod \"344707b1-2bf5-4481-8522-afe6d996c26c\" (UID: \"344707b1-2bf5-4481-8522-afe6d996c26c\") "
Oct 31 14:01:18.988793 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.988668    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-workload-certs" pod "344707b1-2bf5-4481-8522-afe6d996c26c" (UID: "344707b1-2bf5-4481-8522-afe6d996c26c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:18.988793 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.988687    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-workload-socket" pod "344707b1-2bf5-4481-8522-afe6d996c26c" (UID: "344707b1-2bf5-4481-8522-afe6d996c26c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:18.988951 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.988838    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-istio-data" pod "344707b1-2bf5-4481-8522-afe6d996c26c" (UID: "344707b1-2bf5-4481-8522-afe6d996c26c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:18.988951 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.988911    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-credential-socket" pod "344707b1-2bf5-4481-8522-afe6d996c26c" (UID: "344707b1-2bf5-4481-8522-afe6d996c26c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:18.989415 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.989394    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:18.989415 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.989416    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:18.989552 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.989429    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:18.989552 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.989441    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344707b1-2bf5-4481-8522-afe6d996c26c-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:18.989552 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.989453    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:18.990597 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.990571    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-istio-envoy" pod "344707b1-2bf5-4481-8522-afe6d996c26c" (UID: "344707b1-2bf5-4481-8522-afe6d996c26c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:18.990840 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.990822    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344707b1-2bf5-4481-8522-afe6d996c26c-istio-token" pod "344707b1-2bf5-4481-8522-afe6d996c26c" (UID: "344707b1-2bf5-4481-8522-afe6d996c26c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:18.991044 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.991012    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/344707b1-2bf5-4481-8522-afe6d996c26c-istio-podinfo" pod "344707b1-2bf5-4481-8522-afe6d996c26c" (UID: "344707b1-2bf5-4481-8522-afe6d996c26c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:01:18.991135 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:18.991055    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344707b1-2bf5-4481-8522-afe6d996c26c-kube-api-access-hhxmf" pod "344707b1-2bf5-4481-8522-afe6d996c26c" (UID: "344707b1-2bf5-4481-8522-afe6d996c26c"). InnerVolumeSpecName "kube-api-access-hhxmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:19.090862 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:19.090829    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344707b1-2bf5-4481-8522-afe6d996c26c-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:19.090862 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:19.090861    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344707b1-2bf5-4481-8522-afe6d996c26c-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:19.091078 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:19.090873    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhxmf\" (UniqueName: \"kubernetes.io/projected/344707b1-2bf5-4481-8522-afe6d996c26c-kube-api-access-hhxmf\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:19.091078 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:19.090883    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344707b1-2bf5-4481-8522-afe6d996c26c-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:19.707133 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:19.707099    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc" event={"ID":"344707b1-2bf5-4481-8522-afe6d996c26c","Type":"ContainerDied","Data":"f898efaf12aa9e1a006e9239e08e0711cbc617fc8aae342129b0851b982ae9f7"}
Oct 31 14:01:19.707633 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:19.707109    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"
Oct 31 14:01:19.707633 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:19.707169    4660 scope.go:122] "RemoveContainer" containerID="84460f572da86226085fdde211fd222a3ab7b0fb7d4c7d523533ef909b60cb9b"
Oct 31 14:01:19.743508 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:19.743468    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"]
Oct 31 14:01:19.749720 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:19.749689    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hl6bc"]
Oct 31 14:01:20.530193 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:20.530158    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="344707b1-2bf5-4481-8522-afe6d996c26c" path="/var/lib/kubelet/pods/344707b1-2bf5-4481-8522-afe6d996c26c/volumes"
Oct 31 14:01:20.530773 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:20.530749    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f82f37eb-0856-4af8-857b-b855316ee6b0" path="/var/lib/kubelet/pods/f82f37eb-0856-4af8-857b-b855316ee6b0/volumes"
Oct 31 14:01:20.738926 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:01:20.738884    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:01:21.832378 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:21.832338    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"]
Oct 31 14:01:21.832868 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:21.832606    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw" podUID="e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9" containerName="maas-api" containerID="cri-o://b5f9ca46310a1147437a4c1ae1f6f1eadccc8356d9020450945153c94420280e" gracePeriod=30
Oct 31 14:01:22.366762 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.366730    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 14:01:22.407932 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.407902    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-kube-api-access-fzxhk\" (UniqueName: \"kubernetes.io/projected/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-kube-api-access-fzxhk\") pod \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") "
Oct 31 14:01:22.408126 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.407955    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-metrics-certs\") pod \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") "
Oct 31 14:01:22.408126 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.408047    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-openshift-service-ca\") pod \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") "
Oct 31 14:01:22.408126 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.408112    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-maas-api-tls\") pod \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\" (UID: \"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9\") "
Oct 31 14:01:22.408749 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.408715    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-openshift-service-ca" pod "e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9" (UID: "e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:22.410925 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.410897    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-metrics-certs" pod "e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9" (UID: "e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:22.411091 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.410990    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-maas-api-tls" pod "e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9" (UID: "e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:22.411508 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.411471    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-kube-api-access-fzxhk" pod "e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9" (UID: "e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9"). InnerVolumeSpecName "kube-api-access-fzxhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:22.509480 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.509445    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:22.509480 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.509480    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:22.509675 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.509493    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzxhk\" (UniqueName: \"kubernetes.io/projected/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-kube-api-access-fzxhk\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:22.509675 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.509514    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:22.737545 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.737504    4660 generic.go:356] "Generic (PLEG): container finished" podID="e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9" containerID="b5f9ca46310a1147437a4c1ae1f6f1eadccc8356d9020450945153c94420280e" exitCode=0
Oct 31 14:01:22.737722 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.737547    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw" event={"ID":"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9","Type":"ContainerDied","Data":"b5f9ca46310a1147437a4c1ae1f6f1eadccc8356d9020450945153c94420280e"}
Oct 31 14:01:22.737722 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.737567    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"
Oct 31 14:01:22.737722 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.737603    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw" event={"ID":"e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9","Type":"ContainerDied","Data":"5a8419c27d2d81bfa71355c4fe7cc4a2c018f88832703d4f4e23c77992bf30e4"}
Oct 31 14:01:22.737722 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.737630    4660 scope.go:122] "RemoveContainer" containerID="b5f9ca46310a1147437a4c1ae1f6f1eadccc8356d9020450945153c94420280e"
Oct 31 14:01:22.750550 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.750516    4660 scope.go:122] "RemoveContainer" containerID="b5f9ca46310a1147437a4c1ae1f6f1eadccc8356d9020450945153c94420280e"
Oct 31 14:01:22.750840 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:01:22.750807    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f9ca46310a1147437a4c1ae1f6f1eadccc8356d9020450945153c94420280e\": container with ID starting with b5f9ca46310a1147437a4c1ae1f6f1eadccc8356d9020450945153c94420280e not found: ID does not exist" containerID="b5f9ca46310a1147437a4c1ae1f6f1eadccc8356d9020450945153c94420280e"
Oct 31 14:01:22.750899 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.750850    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f9ca46310a1147437a4c1ae1f6f1eadccc8356d9020450945153c94420280e"} err="failed to get container status \"b5f9ca46310a1147437a4c1ae1f6f1eadccc8356d9020450945153c94420280e\": rpc error: code = NotFound desc = could not find container \"b5f9ca46310a1147437a4c1ae1f6f1eadccc8356d9020450945153c94420280e\": container with ID starting with b5f9ca46310a1147437a4c1ae1f6f1eadccc8356d9020450945153c94420280e not found: ID does not exist"
Oct 31 14:01:22.778412 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.778389    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"]
Oct 31 14:01:22.788107 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:22.788081    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-599291-74b8bcbbbc-x7fpw"]
Oct 31 14:01:24.527962 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:24.527931    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9" path="/var/lib/kubelet/pods/e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9/volumes"
Oct 31 14:01:25.643144 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:25.643061    4660 scope.go:122] "RemoveContainer" containerID="c14a75a5800897db4cb158c19a4934bd8f28d5116a168393f1a3cc6c4958ad40"
Oct 31 14:01:32.392082 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:32.391889    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"]
Oct 31 14:01:32.392646 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:32.392204    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn" podUID="21f3684e-26dd-493c-af92-89e1f8b2b18a" containerName="maas-api" containerID="cri-o://b4ab240a68ebc8c7417c999d991182d4c010ee9cd190d54b303ba568fad18fd2" gracePeriod=30
Oct 31 14:01:32.883642 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:32.882416    4660 generic.go:356] "Generic (PLEG): container finished" podID="21f3684e-26dd-493c-af92-89e1f8b2b18a" containerID="b4ab240a68ebc8c7417c999d991182d4c010ee9cd190d54b303ba568fad18fd2" exitCode=0
Oct 31 14:01:32.883642 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:32.882479    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn" event={"ID":"21f3684e-26dd-493c-af92-89e1f8b2b18a","Type":"ContainerDied","Data":"b4ab240a68ebc8c7417c999d991182d4c010ee9cd190d54b303ba568fad18fd2"}
Oct 31 14:01:32.961195 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:32.961168    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 14:01:33.015799 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.015760    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21f3684e-26dd-493c-af92-89e1f8b2b18a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f3684e-26dd-493c-af92-89e1f8b2b18a-openshift-service-ca\") pod \"21f3684e-26dd-493c-af92-89e1f8b2b18a\" (UID: \"21f3684e-26dd-493c-af92-89e1f8b2b18a\") "
Oct 31 14:01:33.015974 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.015859    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f3684e-26dd-493c-af92-89e1f8b2b18a-kube-api-access-wxv94\" (UniqueName: \"kubernetes.io/projected/21f3684e-26dd-493c-af92-89e1f8b2b18a-kube-api-access-wxv94\") pod \"21f3684e-26dd-493c-af92-89e1f8b2b18a\" (UID: \"21f3684e-26dd-493c-af92-89e1f8b2b18a\") "
Oct 31 14:01:33.015974 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.015903    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21f3684e-26dd-493c-af92-89e1f8b2b18a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f3684e-26dd-493c-af92-89e1f8b2b18a-maas-api-tls\") pod \"21f3684e-26dd-493c-af92-89e1f8b2b18a\" (UID: \"21f3684e-26dd-493c-af92-89e1f8b2b18a\") "
Oct 31 14:01:33.016230 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.016200    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f3684e-26dd-493c-af92-89e1f8b2b18a-openshift-service-ca" pod "21f3684e-26dd-493c-af92-89e1f8b2b18a" (UID: "21f3684e-26dd-493c-af92-89e1f8b2b18a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:33.016326 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.016304    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21f3684e-26dd-493c-af92-89e1f8b2b18a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f3684e-26dd-493c-af92-89e1f8b2b18a-metrics-certs\") pod \"21f3684e-26dd-493c-af92-89e1f8b2b18a\" (UID: \"21f3684e-26dd-493c-af92-89e1f8b2b18a\") "
Oct 31 14:01:33.016931 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.016907    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f3684e-26dd-493c-af92-89e1f8b2b18a-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:33.020329 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.020287    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f3684e-26dd-493c-af92-89e1f8b2b18a-kube-api-access-wxv94" pod "21f3684e-26dd-493c-af92-89e1f8b2b18a" (UID: "21f3684e-26dd-493c-af92-89e1f8b2b18a"). InnerVolumeSpecName "kube-api-access-wxv94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:33.020729 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.020706    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21f3684e-26dd-493c-af92-89e1f8b2b18a-maas-api-tls" pod "21f3684e-26dd-493c-af92-89e1f8b2b18a" (UID: "21f3684e-26dd-493c-af92-89e1f8b2b18a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:33.026354 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.026326    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21f3684e-26dd-493c-af92-89e1f8b2b18a-metrics-certs" pod "21f3684e-26dd-493c-af92-89e1f8b2b18a" (UID: "21f3684e-26dd-493c-af92-89e1f8b2b18a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:33.117949 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.117915    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxv94\" (UniqueName: \"kubernetes.io/projected/21f3684e-26dd-493c-af92-89e1f8b2b18a-kube-api-access-wxv94\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:33.117949 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.117947    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f3684e-26dd-493c-af92-89e1f8b2b18a-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:33.118189 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.117961    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f3684e-26dd-493c-af92-89e1f8b2b18a-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:33.890283 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.890248    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn" event={"ID":"21f3684e-26dd-493c-af92-89e1f8b2b18a","Type":"ContainerDied","Data":"2d4e5b803a02d06ac34cede84d2ec4cb8ab95c99368c258712a3fb2ed4c96a61"}
Oct 31 14:01:33.890283 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.890268    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"
Oct 31 14:01:33.890816 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.890347    4660 scope.go:122] "RemoveContainer" containerID="b4ab240a68ebc8c7417c999d991182d4c010ee9cd190d54b303ba568fad18fd2"
Oct 31 14:01:33.920272 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.920236    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"]
Oct 31 14:01:33.926229 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:33.926199    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-403dff-6bdbb959fb-ghtmn"]
Oct 31 14:01:34.531511 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:34.531471    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21f3684e-26dd-493c-af92-89e1f8b2b18a" path="/var/lib/kubelet/pods/21f3684e-26dd-493c-af92-89e1f8b2b18a/volumes"
Oct 31 14:01:35.931483 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:35.931443    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"]
Oct 31 14:01:42.333170 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:42.333134    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"]
Oct 31 14:01:42.333719 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:42.333460    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" containerName="istio-proxy" containerID="cri-o://85cbf17b3968a7dec0fbd0f1d08c0d8c9de8017188454e241c5a3a5a507540ab" gracePeriod=30
Oct 31 14:01:44.638783 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:44.638748    4660 patch_prober.go:29] interesting pod/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:01:44.639177 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:44.638821    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:01:47.605382 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.605311    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 14:01:47.795187 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.795150    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-workload-socket\") pod \"be5b97ff-dbf0-4a26-835f-702598486d61\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") "
Oct 31 14:01:47.795440 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.795201    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be5b97ff-dbf0-4a26-835f-702598486d61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5b97ff-dbf0-4a26-835f-702598486d61-istio-podinfo\") pod \"be5b97ff-dbf0-4a26-835f-702598486d61\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") "
Oct 31 14:01:47.795440 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.795330    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be5b97ff-dbf0-4a26-835f-702598486d61-kube-api-access-2jjtj\" (UniqueName: \"kubernetes.io/projected/be5b97ff-dbf0-4a26-835f-702598486d61-kube-api-access-2jjtj\") pod \"be5b97ff-dbf0-4a26-835f-702598486d61\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") "
Oct 31 14:01:47.795440 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.795370    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-istio-envoy\") pod \"be5b97ff-dbf0-4a26-835f-702598486d61\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") "
Oct 31 14:01:47.795440 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.795428    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-istio-data\") pod \"be5b97ff-dbf0-4a26-835f-702598486d61\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") "
Oct 31 14:01:47.795692 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.795467    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-workload-certs\") pod \"be5b97ff-dbf0-4a26-835f-702598486d61\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") "
Oct 31 14:01:47.795692 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.795521    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be5b97ff-dbf0-4a26-835f-702598486d61-istio-token\" (UniqueName: \"kubernetes.io/projected/be5b97ff-dbf0-4a26-835f-702598486d61-istio-token\") pod \"be5b97ff-dbf0-4a26-835f-702598486d61\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") "
Oct 31 14:01:47.795692 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.795615    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-workload-socket" pod "be5b97ff-dbf0-4a26-835f-702598486d61" (UID: "be5b97ff-dbf0-4a26-835f-702598486d61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:47.795692 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.795661    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be5b97ff-dbf0-4a26-835f-702598486d61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5b97ff-dbf0-4a26-835f-702598486d61-istiod-ca-cert\") pod \"be5b97ff-dbf0-4a26-835f-702598486d61\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") "
Oct 31 14:01:47.795911 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.795740    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-credential-socket\") pod \"be5b97ff-dbf0-4a26-835f-702598486d61\" (UID: \"be5b97ff-dbf0-4a26-835f-702598486d61\") "
Oct 31 14:01:47.795911 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.795846    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-istio-data" pod "be5b97ff-dbf0-4a26-835f-702598486d61" (UID: "be5b97ff-dbf0-4a26-835f-702598486d61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:47.795911 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.795864    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-workload-certs" pod "be5b97ff-dbf0-4a26-835f-702598486d61" (UID: "be5b97ff-dbf0-4a26-835f-702598486d61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:47.796155 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.796132    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be5b97ff-dbf0-4a26-835f-702598486d61-istiod-ca-cert" pod "be5b97ff-dbf0-4a26-835f-702598486d61" (UID: "be5b97ff-dbf0-4a26-835f-702598486d61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:47.796466 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.796353    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-credential-socket" pod "be5b97ff-dbf0-4a26-835f-702598486d61" (UID: "be5b97ff-dbf0-4a26-835f-702598486d61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:47.796466 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.796358    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:47.796466 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.796407    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:47.796466 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.796425    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5b97ff-dbf0-4a26-835f-702598486d61-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:47.796466 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.796440    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:47.797795 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.797765    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be5b97ff-dbf0-4a26-835f-702598486d61-istio-podinfo" pod "be5b97ff-dbf0-4a26-835f-702598486d61" (UID: "be5b97ff-dbf0-4a26-835f-702598486d61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:01:47.797948 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.797929    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5b97ff-dbf0-4a26-835f-702598486d61-istio-token" pod "be5b97ff-dbf0-4a26-835f-702598486d61" (UID: "be5b97ff-dbf0-4a26-835f-702598486d61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:47.798019 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.797999    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-istio-envoy" pod "be5b97ff-dbf0-4a26-835f-702598486d61" (UID: "be5b97ff-dbf0-4a26-835f-702598486d61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:47.798085 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.798066    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5b97ff-dbf0-4a26-835f-702598486d61-kube-api-access-2jjtj" pod "be5b97ff-dbf0-4a26-835f-702598486d61" (UID: "be5b97ff-dbf0-4a26-835f-702598486d61"). InnerVolumeSpecName "kube-api-access-2jjtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:47.897718 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.897677    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jjtj\" (UniqueName: \"kubernetes.io/projected/be5b97ff-dbf0-4a26-835f-702598486d61-kube-api-access-2jjtj\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:47.897718 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.897713    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:47.897718 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.897723    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5b97ff-dbf0-4a26-835f-702598486d61-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:47.897996 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.897735    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b97ff-dbf0-4a26-835f-702598486d61-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:47.897996 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.897746    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5b97ff-dbf0-4a26-835f-702598486d61-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:47.999442 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.999401    4660 generic.go:356] "Generic (PLEG): container finished" podID="be5b97ff-dbf0-4a26-835f-702598486d61" containerID="85cbf17b3968a7dec0fbd0f1d08c0d8c9de8017188454e241c5a3a5a507540ab" exitCode=0
Oct 31 14:01:47.999627 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.999466    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"
Oct 31 14:01:47.999627 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.999490    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" event={"ID":"be5b97ff-dbf0-4a26-835f-702598486d61","Type":"ContainerDied","Data":"85cbf17b3968a7dec0fbd0f1d08c0d8c9de8017188454e241c5a3a5a507540ab"}
Oct 31 14:01:47.999627 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.999589    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f" event={"ID":"be5b97ff-dbf0-4a26-835f-702598486d61","Type":"ContainerDied","Data":"21e79708ca88a752397a9e500d112023675283a1f7a946bacebc7c3482290c5d"}
Oct 31 14:01:47.999627 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:47.999619    4660 scope.go:122] "RemoveContainer" containerID="85cbf17b3968a7dec0fbd0f1d08c0d8c9de8017188454e241c5a3a5a507540ab"
Oct 31 14:01:48.025695 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:48.025664    4660 scope.go:122] "RemoveContainer" containerID="98eea87a15642123c9cb95dada309db6c76f7df967d07c7089babebf9355452b"
Oct 31 14:01:48.046204 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:48.046161    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"]
Oct 31 14:01:48.049758 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:48.049724    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-403dff-openshift-default-7656f86cf5-4c46f"]
Oct 31 14:01:48.056836 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:48.056810    4660 scope.go:122] "RemoveContainer" containerID="85cbf17b3968a7dec0fbd0f1d08c0d8c9de8017188454e241c5a3a5a507540ab"
Oct 31 14:01:48.057221 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:01:48.057197    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85cbf17b3968a7dec0fbd0f1d08c0d8c9de8017188454e241c5a3a5a507540ab\": container with ID starting with 85cbf17b3968a7dec0fbd0f1d08c0d8c9de8017188454e241c5a3a5a507540ab not found: ID does not exist" containerID="85cbf17b3968a7dec0fbd0f1d08c0d8c9de8017188454e241c5a3a5a507540ab"
Oct 31 14:01:48.057342 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:48.057231    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85cbf17b3968a7dec0fbd0f1d08c0d8c9de8017188454e241c5a3a5a507540ab"} err="failed to get container status \"85cbf17b3968a7dec0fbd0f1d08c0d8c9de8017188454e241c5a3a5a507540ab\": rpc error: code = NotFound desc = could not find container \"85cbf17b3968a7dec0fbd0f1d08c0d8c9de8017188454e241c5a3a5a507540ab\": container with ID starting with 85cbf17b3968a7dec0fbd0f1d08c0d8c9de8017188454e241c5a3a5a507540ab not found: ID does not exist"
Oct 31 14:01:48.057342 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:48.057252    4660 scope.go:122] "RemoveContainer" containerID="98eea87a15642123c9cb95dada309db6c76f7df967d07c7089babebf9355452b"
Oct 31 14:01:48.057638 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:01:48.057608    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98eea87a15642123c9cb95dada309db6c76f7df967d07c7089babebf9355452b\": container with ID starting with 98eea87a15642123c9cb95dada309db6c76f7df967d07c7089babebf9355452b not found: ID does not exist" containerID="98eea87a15642123c9cb95dada309db6c76f7df967d07c7089babebf9355452b"
Oct 31 14:01:48.057742 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:48.057648    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98eea87a15642123c9cb95dada309db6c76f7df967d07c7089babebf9355452b"} err="failed to get container status \"98eea87a15642123c9cb95dada309db6c76f7df967d07c7089babebf9355452b\": rpc error: code = NotFound desc = could not find container \"98eea87a15642123c9cb95dada309db6c76f7df967d07c7089babebf9355452b\": container with ID starting with 98eea87a15642123c9cb95dada309db6c76f7df967d07c7089babebf9355452b not found: ID does not exist"
Oct 31 14:01:48.528093 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:48.528057    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" path="/var/lib/kubelet/pods/be5b97ff-dbf0-4a26-835f-702598486d61/volumes"
Oct 31 14:01:50.734583 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:01:50.734543    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:01:50.961146 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:50.961101    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main" containerID="cri-o://7eef1bf5bae5166cbb033c5bfda43dea92d779cbd0f7c7ef4770776fc071ede5" gracePeriod=45
Oct 31 14:01:51.257399 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.257370    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 14:01:51.353164 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.352430    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4bb048a-1211-4eea-b483-f1c22a9da6a5-kube-api-access-kbjrt\" (UniqueName: \"kubernetes.io/projected/d4bb048a-1211-4eea-b483-f1c22a9da6a5-kube-api-access-kbjrt\") pod \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") "
Oct 31 14:01:51.353164 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.352536    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-model-cache\") pod \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") "
Oct 31 14:01:51.353164 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.352574    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4bb048a-1211-4eea-b483-f1c22a9da6a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4bb048a-1211-4eea-b483-f1c22a9da6a5-tls-certs\") pod \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") "
Oct 31 14:01:51.353164 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.352732    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-dshm\") pod \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") "
Oct 31 14:01:51.353164 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.352799    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-tmp-dir\") pod \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") "
Oct 31 14:01:51.353164 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.352859    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-home\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-home\") pod \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\" (UID: \"d4bb048a-1211-4eea-b483-f1c22a9da6a5\") "
Oct 31 14:01:51.353164 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.352864    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-model-cache" pod "d4bb048a-1211-4eea-b483-f1c22a9da6a5" (UID: "d4bb048a-1211-4eea-b483-f1c22a9da6a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:51.353782 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.353444    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:51.353782 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.353719    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-tmp-dir" pod "d4bb048a-1211-4eea-b483-f1c22a9da6a5" (UID: "d4bb048a-1211-4eea-b483-f1c22a9da6a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:51.353999 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.353962    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-home" pod "d4bb048a-1211-4eea-b483-f1c22a9da6a5" (UID: "d4bb048a-1211-4eea-b483-f1c22a9da6a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:51.357878 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.357816    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4bb048a-1211-4eea-b483-f1c22a9da6a5-kube-api-access-kbjrt" pod "d4bb048a-1211-4eea-b483-f1c22a9da6a5" (UID: "d4bb048a-1211-4eea-b483-f1c22a9da6a5"). InnerVolumeSpecName "kube-api-access-kbjrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:51.357984 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.357936    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-dshm" pod "d4bb048a-1211-4eea-b483-f1c22a9da6a5" (UID: "d4bb048a-1211-4eea-b483-f1c22a9da6a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:51.362932 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.362909    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4bb048a-1211-4eea-b483-f1c22a9da6a5-tls-certs" pod "d4bb048a-1211-4eea-b483-f1c22a9da6a5" (UID: "d4bb048a-1211-4eea-b483-f1c22a9da6a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:51.454802 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.454762    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4bb048a-1211-4eea-b483-f1c22a9da6a5-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:51.454802 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.454798    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:51.454802 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.454808    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:51.455058 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.454817    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4bb048a-1211-4eea-b483-f1c22a9da6a5-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:51.455058 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:51.454825    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbjrt\" (UniqueName: \"kubernetes.io/projected/d4bb048a-1211-4eea-b483-f1c22a9da6a5-kube-api-access-kbjrt\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:52.040435 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:52.040392    4660 generic.go:356] "Generic (PLEG): container finished" podID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerID="7eef1bf5bae5166cbb033c5bfda43dea92d779cbd0f7c7ef4770776fc071ede5" exitCode=0
Oct 31 14:01:52.040874 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:52.040449    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"
Oct 31 14:01:52.040874 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:52.040480    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" event={"ID":"d4bb048a-1211-4eea-b483-f1c22a9da6a5","Type":"ContainerDied","Data":"7eef1bf5bae5166cbb033c5bfda43dea92d779cbd0f7c7ef4770776fc071ede5"}
Oct 31 14:01:52.040874 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:52.040528    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm" event={"ID":"d4bb048a-1211-4eea-b483-f1c22a9da6a5","Type":"ContainerDied","Data":"a5a4e5dee28d3448d992d74ae5f29ad05db9eaef79d17041e158737fb67230ef"}
Oct 31 14:01:52.040874 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:52.040545    4660 scope.go:122] "RemoveContainer" containerID="7eef1bf5bae5166cbb033c5bfda43dea92d779cbd0f7c7ef4770776fc071ede5"
Oct 31 14:01:52.052175 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:52.052154    4660 scope.go:122] "RemoveContainer" containerID="7eef1bf5bae5166cbb033c5bfda43dea92d779cbd0f7c7ef4770776fc071ede5"
Oct 31 14:01:52.052534 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:01:52.052512    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eef1bf5bae5166cbb033c5bfda43dea92d779cbd0f7c7ef4770776fc071ede5\": container with ID starting with 7eef1bf5bae5166cbb033c5bfda43dea92d779cbd0f7c7ef4770776fc071ede5 not found: ID does not exist" containerID="7eef1bf5bae5166cbb033c5bfda43dea92d779cbd0f7c7ef4770776fc071ede5"
Oct 31 14:01:52.052624 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:52.052541    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eef1bf5bae5166cbb033c5bfda43dea92d779cbd0f7c7ef4770776fc071ede5"} err="failed to get container status \"7eef1bf5bae5166cbb033c5bfda43dea92d779cbd0f7c7ef4770776fc071ede5\": rpc error: code = NotFound desc = could not find container \"7eef1bf5bae5166cbb033c5bfda43dea92d779cbd0f7c7ef4770776fc071ede5\": container with ID starting with 7eef1bf5bae5166cbb033c5bfda43dea92d779cbd0f7c7ef4770776fc071ede5 not found: ID does not exist"
Oct 31 14:01:52.068072 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:52.068037    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"]
Oct 31 14:01:52.074804 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:52.074774    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/distinct-2-w1-d85d82-kserve-7f59c59d4d-mmtbm"]
Oct 31 14:01:52.529142 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:52.529108    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" path="/var/lib/kubelet/pods/d4bb048a-1211-4eea-b483-f1c22a9da6a5/volumes"
Oct 31 14:01:54.123551 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:54.123509    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"]
Oct 31 14:01:54.124032 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:54.123740    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn" podUID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerName="istio-proxy" containerID="cri-o://54424b617f2a42315c52130074f9dbdf199804907190e469deea62624f056a22" gracePeriod=30
Oct 31 14:01:58.128216 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:58.128170    4660 patch_prober.go:29] interesting pod/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:01:58.128769 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:58.128247    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn" podUID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:01:58.552581 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:58.550852    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"]
Oct 31 14:01:58.552581 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:58.551117    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb" podUID="de5ecce4-e294-4623-b83d-9ba9a5e74bca" containerName="maas-api" containerID="cri-o://5c03feec4ad1a7411b0718842378ae7eee8fc9d76ca7743ca0895aa990bee7d8" gracePeriod=30
Oct 31 14:01:58.709912 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:58.709854    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"]
Oct 31 14:01:58.710594 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:58.710480    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw" podUID="533c7ae9-3ae7-4cd4-a38f-98c35110b250" containerName="maas-api" containerID="cri-o://61222bc638035d0b17b0786b616c72fabbadde58638182cc0be00cbe74e85066" gracePeriod=30
Oct 31 14:01:58.737453 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:58.736441    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw" podUID="533c7ae9-3ae7-4cd4-a38f-98c35110b250" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": dial tcp 10.133.0.118:8443: connect: connection refused"
Oct 31 14:01:59.126240 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.125912    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 14:01:59.154192 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.142719    4660 generic.go:356] "Generic (PLEG): container finished" podID="de5ecce4-e294-4623-b83d-9ba9a5e74bca" containerID="5c03feec4ad1a7411b0718842378ae7eee8fc9d76ca7743ca0895aa990bee7d8" exitCode=0
Oct 31 14:01:59.154192 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.142921    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb" event={"ID":"de5ecce4-e294-4623-b83d-9ba9a5e74bca","Type":"ContainerDied","Data":"5c03feec4ad1a7411b0718842378ae7eee8fc9d76ca7743ca0895aa990bee7d8"}
Oct 31 14:01:59.154192 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.142974    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb" event={"ID":"de5ecce4-e294-4623-b83d-9ba9a5e74bca","Type":"ContainerDied","Data":"6424d1935cf8a100c4a64ef226ca2353195dc806b510b5dd2c3cb732588eb3d1"}
Oct 31 14:01:59.154192 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.143004    4660 scope.go:122] "RemoveContainer" containerID="5c03feec4ad1a7411b0718842378ae7eee8fc9d76ca7743ca0895aa990bee7d8"
Oct 31 14:01:59.154192 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.143170    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"
Oct 31 14:01:59.166034 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.165137    4660 generic.go:356] "Generic (PLEG): container finished" podID="533c7ae9-3ae7-4cd4-a38f-98c35110b250" containerID="61222bc638035d0b17b0786b616c72fabbadde58638182cc0be00cbe74e85066" exitCode=0
Oct 31 14:01:59.166034 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.165270    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw" event={"ID":"533c7ae9-3ae7-4cd4-a38f-98c35110b250","Type":"ContainerDied","Data":"61222bc638035d0b17b0786b616c72fabbadde58638182cc0be00cbe74e85066"}
Oct 31 14:01:59.235362 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.225506    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de5ecce4-e294-4623-b83d-9ba9a5e74bca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de5ecce4-e294-4623-b83d-9ba9a5e74bca-maas-api-tls\") pod \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\" (UID: \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\") "
Oct 31 14:01:59.235362 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.225696    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5ecce4-e294-4623-b83d-9ba9a5e74bca-kube-api-access-9dr2s\" (UniqueName: \"kubernetes.io/projected/de5ecce4-e294-4623-b83d-9ba9a5e74bca-kube-api-access-9dr2s\") pod \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\" (UID: \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\") "
Oct 31 14:01:59.235362 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.225807    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de5ecce4-e294-4623-b83d-9ba9a5e74bca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5ecce4-e294-4623-b83d-9ba9a5e74bca-metrics-certs\") pod \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\" (UID: \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\") "
Oct 31 14:01:59.235362 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.225859    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de5ecce4-e294-4623-b83d-9ba9a5e74bca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5ecce4-e294-4623-b83d-9ba9a5e74bca-openshift-service-ca\") pod \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\" (UID: \"de5ecce4-e294-4623-b83d-9ba9a5e74bca\") "
Oct 31 14:01:59.235362 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.226819    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5ecce4-e294-4623-b83d-9ba9a5e74bca-openshift-service-ca" pod "de5ecce4-e294-4623-b83d-9ba9a5e74bca" (UID: "de5ecce4-e294-4623-b83d-9ba9a5e74bca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:59.242446 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.242375    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de5ecce4-e294-4623-b83d-9ba9a5e74bca-metrics-certs" pod "de5ecce4-e294-4623-b83d-9ba9a5e74bca" (UID: "de5ecce4-e294-4623-b83d-9ba9a5e74bca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:59.242846 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.242804    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de5ecce4-e294-4623-b83d-9ba9a5e74bca-maas-api-tls" pod "de5ecce4-e294-4623-b83d-9ba9a5e74bca" (UID: "de5ecce4-e294-4623-b83d-9ba9a5e74bca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:59.245736 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.245703    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5ecce4-e294-4623-b83d-9ba9a5e74bca-kube-api-access-9dr2s" pod "de5ecce4-e294-4623-b83d-9ba9a5e74bca" (UID: "de5ecce4-e294-4623-b83d-9ba9a5e74bca"). InnerVolumeSpecName "kube-api-access-9dr2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:59.261914 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.261858    4660 scope.go:122] "RemoveContainer" containerID="5c03feec4ad1a7411b0718842378ae7eee8fc9d76ca7743ca0895aa990bee7d8"
Oct 31 14:01:59.262584 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:01:59.262550    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c03feec4ad1a7411b0718842378ae7eee8fc9d76ca7743ca0895aa990bee7d8\": container with ID starting with 5c03feec4ad1a7411b0718842378ae7eee8fc9d76ca7743ca0895aa990bee7d8 not found: ID does not exist" containerID="5c03feec4ad1a7411b0718842378ae7eee8fc9d76ca7743ca0895aa990bee7d8"
Oct 31 14:01:59.262696 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.262602    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c03feec4ad1a7411b0718842378ae7eee8fc9d76ca7743ca0895aa990bee7d8"} err="failed to get container status \"5c03feec4ad1a7411b0718842378ae7eee8fc9d76ca7743ca0895aa990bee7d8\": rpc error: code = NotFound desc = could not find container \"5c03feec4ad1a7411b0718842378ae7eee8fc9d76ca7743ca0895aa990bee7d8\": container with ID starting with 5c03feec4ad1a7411b0718842378ae7eee8fc9d76ca7743ca0895aa990bee7d8 not found: ID does not exist"
Oct 31 14:01:59.330938 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.330828    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dr2s\" (UniqueName: \"kubernetes.io/projected/de5ecce4-e294-4623-b83d-9ba9a5e74bca-kube-api-access-9dr2s\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:59.331184 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.331162    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5ecce4-e294-4623-b83d-9ba9a5e74bca-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:59.331313 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.331289    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5ecce4-e294-4623-b83d-9ba9a5e74bca-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:59.331426 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.331414    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de5ecce4-e294-4623-b83d-9ba9a5e74bca-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:59.370603 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.370201    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 14:01:59.443494 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.443457    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533c7ae9-3ae7-4cd4-a38f-98c35110b250-kube-api-access-ltvvd\" (UniqueName: \"kubernetes.io/projected/533c7ae9-3ae7-4cd4-a38f-98c35110b250-kube-api-access-ltvvd\") pod \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\" (UID: \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\") "
Oct 31 14:01:59.443866 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.443841    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/533c7ae9-3ae7-4cd4-a38f-98c35110b250-metrics-certs\" (UniqueName: \"kubernetes.io/secret/533c7ae9-3ae7-4cd4-a38f-98c35110b250-metrics-certs\") pod \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\" (UID: \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\") "
Oct 31 14:01:59.470989 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.470938    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533c7ae9-3ae7-4cd4-a38f-98c35110b250-metrics-certs" pod "533c7ae9-3ae7-4cd4-a38f-98c35110b250" (UID: "533c7ae9-3ae7-4cd4-a38f-98c35110b250"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:59.473416 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.473355    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533c7ae9-3ae7-4cd4-a38f-98c35110b250-kube-api-access-ltvvd" pod "533c7ae9-3ae7-4cd4-a38f-98c35110b250" (UID: "533c7ae9-3ae7-4cd4-a38f-98c35110b250"). InnerVolumeSpecName "kube-api-access-ltvvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:59.491470 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.491433    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"]
Oct 31 14:01:59.497440 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.497412    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 14:01:59.498169 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.498141    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-14ba5a-689c69b898-vd2bb"]
Oct 31 14:01:59.545074 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.545035    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-data\") pod \"e71b679c-5dff-4c19-bef4-a4bf19093b34\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") "
Oct 31 14:01:59.545823 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.545781    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-workload-socket\") pod \"e71b679c-5dff-4c19-bef4-a4bf19093b34\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") "
Oct 31 14:01:59.545961 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.545853    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e71b679c-5dff-4c19-bef4-a4bf19093b34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71b679c-5dff-4c19-bef4-a4bf19093b34-istiod-ca-cert\") pod \"e71b679c-5dff-4c19-bef4-a4bf19093b34\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") "
Oct 31 14:01:59.545961 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.545889    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533c7ae9-3ae7-4cd4-a38f-98c35110b250-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533c7ae9-3ae7-4cd4-a38f-98c35110b250-openshift-service-ca\") pod \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\" (UID: \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\") "
Oct 31 14:01:59.545961 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.545925    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-token\" (UniqueName: \"kubernetes.io/projected/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-token\") pod \"e71b679c-5dff-4c19-bef4-a4bf19093b34\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") "
Oct 31 14:01:59.546080 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.546047    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-envoy\") pod \"e71b679c-5dff-4c19-bef4-a4bf19093b34\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") "
Oct 31 14:01:59.546125 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.546084    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-credential-socket\") pod \"e71b679c-5dff-4c19-bef4-a4bf19093b34\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") "
Oct 31 14:01:59.546167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.546130    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/533c7ae9-3ae7-4cd4-a38f-98c35110b250-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533c7ae9-3ae7-4cd4-a38f-98c35110b250-maas-api-tls\") pod \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\" (UID: \"533c7ae9-3ae7-4cd4-a38f-98c35110b250\") "
Oct 31 14:01:59.546239 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.546185    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71b679c-5dff-4c19-bef4-a4bf19093b34-kube-api-access-m97pn\" (UniqueName: \"kubernetes.io/projected/e71b679c-5dff-4c19-bef4-a4bf19093b34-kube-api-access-m97pn\") pod \"e71b679c-5dff-4c19-bef4-a4bf19093b34\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") "
Oct 31 14:01:59.546239 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.546222    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-podinfo\") pod \"e71b679c-5dff-4c19-bef4-a4bf19093b34\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") "
Oct 31 14:01:59.546320 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.546263    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-workload-certs\") pod \"e71b679c-5dff-4c19-bef4-a4bf19093b34\" (UID: \"e71b679c-5dff-4c19-bef4-a4bf19093b34\") "
Oct 31 14:01:59.546975 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.546951    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533c7ae9-3ae7-4cd4-a38f-98c35110b250-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:59.547022 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.546984    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltvvd\" (UniqueName: \"kubernetes.io/projected/533c7ae9-3ae7-4cd4-a38f-98c35110b250-kube-api-access-ltvvd\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:59.548252 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.545670    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-data" pod "e71b679c-5dff-4c19-bef4-a4bf19093b34" (UID: "e71b679c-5dff-4c19-bef4-a4bf19093b34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:59.548252 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.547921    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-credential-socket" pod "e71b679c-5dff-4c19-bef4-a4bf19093b34" (UID: "e71b679c-5dff-4c19-bef4-a4bf19093b34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:59.548252 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.547894    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71b679c-5dff-4c19-bef4-a4bf19093b34-istiod-ca-cert" pod "e71b679c-5dff-4c19-bef4-a4bf19093b34" (UID: "e71b679c-5dff-4c19-bef4-a4bf19093b34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:59.548252 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.548076    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533c7ae9-3ae7-4cd4-a38f-98c35110b250-openshift-service-ca" pod "533c7ae9-3ae7-4cd4-a38f-98c35110b250" (UID: "533c7ae9-3ae7-4cd4-a38f-98c35110b250"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:59.548252 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.548226    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-workload-socket" pod "e71b679c-5dff-4c19-bef4-a4bf19093b34" (UID: "e71b679c-5dff-4c19-bef4-a4bf19093b34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:59.548657 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.548512    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-workload-certs" pod "e71b679c-5dff-4c19-bef4-a4bf19093b34" (UID: "e71b679c-5dff-4c19-bef4-a4bf19093b34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:59.550881 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.550757    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-envoy" pod "e71b679c-5dff-4c19-bef4-a4bf19093b34" (UID: "e71b679c-5dff-4c19-bef4-a4bf19093b34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:59.550881 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.550797    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71b679c-5dff-4c19-bef4-a4bf19093b34-kube-api-access-m97pn" pod "e71b679c-5dff-4c19-bef4-a4bf19093b34" (UID: "e71b679c-5dff-4c19-bef4-a4bf19093b34"). InnerVolumeSpecName "kube-api-access-m97pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:59.550881 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.550846    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533c7ae9-3ae7-4cd4-a38f-98c35110b250-maas-api-tls" pod "533c7ae9-3ae7-4cd4-a38f-98c35110b250" (UID: "533c7ae9-3ae7-4cd4-a38f-98c35110b250"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:59.552382 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.552358    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-token" pod "e71b679c-5dff-4c19-bef4-a4bf19093b34" (UID: "e71b679c-5dff-4c19-bef4-a4bf19093b34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:59.552606 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.552586    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-podinfo" pod "e71b679c-5dff-4c19-bef4-a4bf19093b34" (UID: "e71b679c-5dff-4c19-bef4-a4bf19093b34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:01:59.648517 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.648463    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:59.648517 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.648509    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:59.648517 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.648526    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71b679c-5dff-4c19-bef4-a4bf19093b34-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:59.648860 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.648538    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533c7ae9-3ae7-4cd4-a38f-98c35110b250-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:59.648860 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.648551    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:59.648860 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.648565    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:59.648860 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.648582    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:59.648860 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.648606    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533c7ae9-3ae7-4cd4-a38f-98c35110b250-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:59.648860 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.648619    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m97pn\" (UniqueName: \"kubernetes.io/projected/e71b679c-5dff-4c19-bef4-a4bf19093b34-kube-api-access-m97pn\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:59.648860 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.648634    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71b679c-5dff-4c19-bef4-a4bf19093b34-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:59.648860 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:01:59.648643    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71b679c-5dff-4c19-bef4-a4bf19093b34-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:00.174606 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.174571    4660 generic.go:356] "Generic (PLEG): container finished" podID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerID="54424b617f2a42315c52130074f9dbdf199804907190e469deea62624f056a22" exitCode=0
Oct 31 14:02:00.175100 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.174620    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"
Oct 31 14:02:00.175100 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.174656    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn" event={"ID":"e71b679c-5dff-4c19-bef4-a4bf19093b34","Type":"ContainerDied","Data":"54424b617f2a42315c52130074f9dbdf199804907190e469deea62624f056a22"}
Oct 31 14:02:00.175100 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.174721    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn" event={"ID":"e71b679c-5dff-4c19-bef4-a4bf19093b34","Type":"ContainerDied","Data":"d81a0c2360fbb7671a5535ee44e6ab9383b237881e9ec0d20ce6aa47e4274d0f"}
Oct 31 14:02:00.175100 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.174754    4660 scope.go:122] "RemoveContainer" containerID="54424b617f2a42315c52130074f9dbdf199804907190e469deea62624f056a22"
Oct 31 14:02:00.176201 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.176186    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"
Oct 31 14:02:00.176277 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.176183    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw" event={"ID":"533c7ae9-3ae7-4cd4-a38f-98c35110b250","Type":"ContainerDied","Data":"681ce368deb8e291b685c1c1b719891621b6cab7ce3aa3923949098bf2d3aa84"}
Oct 31 14:02:00.189100 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.189078    4660 scope.go:122] "RemoveContainer" containerID="5f72848048b7bb19b1b96647657945c5c2563c2767c3bb671e6f5d90a79e7d72"
Oct 31 14:02:00.200090 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.200062    4660 scope.go:122] "RemoveContainer" containerID="54424b617f2a42315c52130074f9dbdf199804907190e469deea62624f056a22"
Oct 31 14:02:00.200594 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:02:00.200563    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54424b617f2a42315c52130074f9dbdf199804907190e469deea62624f056a22\": container with ID starting with 54424b617f2a42315c52130074f9dbdf199804907190e469deea62624f056a22 not found: ID does not exist" containerID="54424b617f2a42315c52130074f9dbdf199804907190e469deea62624f056a22"
Oct 31 14:02:00.200820 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.200785    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54424b617f2a42315c52130074f9dbdf199804907190e469deea62624f056a22"} err="failed to get container status \"54424b617f2a42315c52130074f9dbdf199804907190e469deea62624f056a22\": rpc error: code = NotFound desc = could not find container \"54424b617f2a42315c52130074f9dbdf199804907190e469deea62624f056a22\": container with ID starting with 54424b617f2a42315c52130074f9dbdf199804907190e469deea62624f056a22 not found: ID does not exist"
Oct 31 14:02:00.200820 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.200819    4660 scope.go:122] "RemoveContainer" containerID="5f72848048b7bb19b1b96647657945c5c2563c2767c3bb671e6f5d90a79e7d72"
Oct 31 14:02:00.201138 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:02:00.201120    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f72848048b7bb19b1b96647657945c5c2563c2767c3bb671e6f5d90a79e7d72\": container with ID starting with 5f72848048b7bb19b1b96647657945c5c2563c2767c3bb671e6f5d90a79e7d72 not found: ID does not exist" containerID="5f72848048b7bb19b1b96647657945c5c2563c2767c3bb671e6f5d90a79e7d72"
Oct 31 14:02:00.201198 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.201143    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f72848048b7bb19b1b96647657945c5c2563c2767c3bb671e6f5d90a79e7d72"} err="failed to get container status \"5f72848048b7bb19b1b96647657945c5c2563c2767c3bb671e6f5d90a79e7d72\": rpc error: code = NotFound desc = could not find container \"5f72848048b7bb19b1b96647657945c5c2563c2767c3bb671e6f5d90a79e7d72\": container with ID starting with 5f72848048b7bb19b1b96647657945c5c2563c2767c3bb671e6f5d90a79e7d72 not found: ID does not exist"
Oct 31 14:02:00.201198 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.201155    4660 scope.go:122] "RemoveContainer" containerID="61222bc638035d0b17b0786b616c72fabbadde58638182cc0be00cbe74e85066"
Oct 31 14:02:00.210613 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.210584    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"]
Oct 31 14:02:00.214274 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.214246    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-nvdtn"]
Oct 31 14:02:00.227464 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.227426    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"]
Oct 31 14:02:00.234804 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.234770    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-15dc2f-65f49f586f-xbkjw"]
Oct 31 14:02:00.536761 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.536674    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533c7ae9-3ae7-4cd4-a38f-98c35110b250" path="/var/lib/kubelet/pods/533c7ae9-3ae7-4cd4-a38f-98c35110b250/volumes"
Oct 31 14:02:00.537370 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.537350    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de5ecce4-e294-4623-b83d-9ba9a5e74bca" path="/var/lib/kubelet/pods/de5ecce4-e294-4623-b83d-9ba9a5e74bca/volumes"
Oct 31 14:02:00.538002 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:00.537983    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e71b679c-5dff-4c19-bef4-a4bf19093b34" path="/var/lib/kubelet/pods/e71b679c-5dff-4c19-bef4-a4bf19093b34/volumes"
Oct 31 14:02:08.479982 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:08.479938    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"]
Oct 31 14:02:08.480531 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:08.480212    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f" podUID="5db28b27-5dcf-4468-8ebc-951e8f52613f" containerName="istio-proxy" containerID="cri-o://5b5e52e31c0d6f436e75c900fb2e6e8c8d1b5898b895b391d96383255608870a" gracePeriod=30
Oct 31 14:02:09.225677 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:09.225628    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"]
Oct 31 14:02:09.225949 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:09.225898    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9" podUID="fb081f1b-a184-4656-a834-240605f9693b" containerName="istio-proxy" containerID="cri-o://0c543a4a65bbfb2b89063ee6f745227e6863eadf7b0b6fc716a7f44ff86f7558" gracePeriod=30
Oct 31 14:02:10.362053 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:10.362008    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:02:10.362515 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:10.362090    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f" podUID="5db28b27-5dcf-4468-8ebc-951e8f52613f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:02:13.792560 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.792530    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 14:02:13.866858 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.866815    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-workload-socket\") pod \"5db28b27-5dcf-4468-8ebc-951e8f52613f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") "
Oct 31 14:02:13.867852 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.867811    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-podinfo\") pod \"5db28b27-5dcf-4468-8ebc-951e8f52613f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") "
Oct 31 14:02:13.868095 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.868078    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db28b27-5dcf-4468-8ebc-951e8f52613f-kube-api-access-cwn95\" (UniqueName: \"kubernetes.io/projected/5db28b27-5dcf-4468-8ebc-951e8f52613f-kube-api-access-cwn95\") pod \"5db28b27-5dcf-4468-8ebc-951e8f52613f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") "
Oct 31 14:02:13.868261 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.868221    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-data\") pod \"5db28b27-5dcf-4468-8ebc-951e8f52613f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") "
Oct 31 14:02:13.868425 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.868404    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-envoy\") pod \"5db28b27-5dcf-4468-8ebc-951e8f52613f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") "
Oct 31 14:02:13.868598 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.868584    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-token\" (UniqueName: \"kubernetes.io/projected/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-token\") pod \"5db28b27-5dcf-4468-8ebc-951e8f52613f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") "
Oct 31 14:02:13.868767 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.868753    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-credential-socket\") pod \"5db28b27-5dcf-4468-8ebc-951e8f52613f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") "
Oct 31 14:02:13.868881 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.868867    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db28b27-5dcf-4468-8ebc-951e8f52613f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db28b27-5dcf-4468-8ebc-951e8f52613f-istiod-ca-cert\") pod \"5db28b27-5dcf-4468-8ebc-951e8f52613f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") "
Oct 31 14:02:13.868990 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.868975    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-workload-certs\") pod \"5db28b27-5dcf-4468-8ebc-951e8f52613f\" (UID: \"5db28b27-5dcf-4468-8ebc-951e8f52613f\") "
Oct 31 14:02:13.869974 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.869946    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-workload-certs" pod "5db28b27-5dcf-4468-8ebc-951e8f52613f" (UID: "5db28b27-5dcf-4468-8ebc-951e8f52613f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:13.870315 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.870271    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-workload-socket" pod "5db28b27-5dcf-4468-8ebc-951e8f52613f" (UID: "5db28b27-5dcf-4468-8ebc-951e8f52613f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:13.871443 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.871133    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-data" pod "5db28b27-5dcf-4468-8ebc-951e8f52613f" (UID: "5db28b27-5dcf-4468-8ebc-951e8f52613f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:13.872379 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.872339    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db28b27-5dcf-4468-8ebc-951e8f52613f-istiod-ca-cert" pod "5db28b27-5dcf-4468-8ebc-951e8f52613f" (UID: "5db28b27-5dcf-4468-8ebc-951e8f52613f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:13.872797 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.872767    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-credential-socket" pod "5db28b27-5dcf-4468-8ebc-951e8f52613f" (UID: "5db28b27-5dcf-4468-8ebc-951e8f52613f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:13.873008 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.872974    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-envoy" pod "5db28b27-5dcf-4468-8ebc-951e8f52613f" (UID: "5db28b27-5dcf-4468-8ebc-951e8f52613f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:13.874883 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.874856    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-token" pod "5db28b27-5dcf-4468-8ebc-951e8f52613f" (UID: "5db28b27-5dcf-4468-8ebc-951e8f52613f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:13.875546 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.875522    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-podinfo" pod "5db28b27-5dcf-4468-8ebc-951e8f52613f" (UID: "5db28b27-5dcf-4468-8ebc-951e8f52613f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:02:13.877503 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.877474    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db28b27-5dcf-4468-8ebc-951e8f52613f-kube-api-access-cwn95" pod "5db28b27-5dcf-4468-8ebc-951e8f52613f" (UID: "5db28b27-5dcf-4468-8ebc-951e8f52613f"). InnerVolumeSpecName "kube-api-access-cwn95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:13.970533 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.970495    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:13.970533 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.970526    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwn95\" (UniqueName: \"kubernetes.io/projected/5db28b27-5dcf-4468-8ebc-951e8f52613f-kube-api-access-cwn95\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:13.970533 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.970537    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:13.970773 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.970559    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:13.970773 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.970571    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db28b27-5dcf-4468-8ebc-951e8f52613f-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:13.970773 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.970579    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:13.970773 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.970589    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db28b27-5dcf-4468-8ebc-951e8f52613f-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:13.970773 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.970597    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:13.970773 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:13.970606    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db28b27-5dcf-4468-8ebc-951e8f52613f-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:14.053556 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.047698    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"]
Oct 31 14:02:14.181729 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.181696    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"]
Oct 31 14:02:14.368962 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.358424    4660 generic.go:356] "Generic (PLEG): container finished" podID="5db28b27-5dcf-4468-8ebc-951e8f52613f" containerID="5b5e52e31c0d6f436e75c900fb2e6e8c8d1b5898b895b391d96383255608870a" exitCode=0
Oct 31 14:02:14.368962 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.358507    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"
Oct 31 14:02:14.368962 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.358682    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f" event={"ID":"5db28b27-5dcf-4468-8ebc-951e8f52613f","Type":"ContainerDied","Data":"5b5e52e31c0d6f436e75c900fb2e6e8c8d1b5898b895b391d96383255608870a"}
Oct 31 14:02:14.368962 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.358727    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f" event={"ID":"5db28b27-5dcf-4468-8ebc-951e8f52613f","Type":"ContainerDied","Data":"4c8c0364694170285c671c64996bbaf4826114a4002bc95e4f16f2cc4905941c"}
Oct 31 14:02:14.368962 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.358754    4660 scope.go:122] "RemoveContainer" containerID="5b5e52e31c0d6f436e75c900fb2e6e8c8d1b5898b895b391d96383255608870a"
Oct 31 14:02:14.375229 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.375087    4660 generic.go:356] "Generic (PLEG): container finished" podID="fb081f1b-a184-4656-a834-240605f9693b" containerID="0c543a4a65bbfb2b89063ee6f745227e6863eadf7b0b6fc716a7f44ff86f7558" exitCode=0
Oct 31 14:02:14.375229 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.375155    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9" event={"ID":"fb081f1b-a184-4656-a834-240605f9693b","Type":"ContainerDied","Data":"0c543a4a65bbfb2b89063ee6f745227e6863eadf7b0b6fc716a7f44ff86f7558"}
Oct 31 14:02:14.375474 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.375443    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" containerID="cri-o://5857f4768517a5769b30be330ccf13c59e589f0860b7e3bcfdd977e9da83330d" gracePeriod=30
Oct 31 14:02:14.384581 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.384555    4660 scope.go:122] "RemoveContainer" containerID="5b5e52e31c0d6f436e75c900fb2e6e8c8d1b5898b895b391d96383255608870a"
Oct 31 14:02:14.384909 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:02:14.384882    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b5e52e31c0d6f436e75c900fb2e6e8c8d1b5898b895b391d96383255608870a\": container with ID starting with 5b5e52e31c0d6f436e75c900fb2e6e8c8d1b5898b895b391d96383255608870a not found: ID does not exist" containerID="5b5e52e31c0d6f436e75c900fb2e6e8c8d1b5898b895b391d96383255608870a"
Oct 31 14:02:14.385010 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.384918    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5e52e31c0d6f436e75c900fb2e6e8c8d1b5898b895b391d96383255608870a"} err="failed to get container status \"5b5e52e31c0d6f436e75c900fb2e6e8c8d1b5898b895b391d96383255608870a\": rpc error: code = NotFound desc = could not find container \"5b5e52e31c0d6f436e75c900fb2e6e8c8d1b5898b895b391d96383255608870a\": container with ID starting with 5b5e52e31c0d6f436e75c900fb2e6e8c8d1b5898b895b391d96383255608870a not found: ID does not exist"
Oct 31 14:02:14.509075 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.509045    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"]
Oct 31 14:02:14.533386 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.533350    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-6g55f"]
Oct 31 14:02:14.942310 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.942256    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 14:02:14.988426 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.988382    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-credential-socket\") pod \"fb081f1b-a184-4656-a834-240605f9693b\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") "
Oct 31 14:02:14.988630 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.988439    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb081f1b-a184-4656-a834-240605f9693b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb081f1b-a184-4656-a834-240605f9693b-istiod-ca-cert\") pod \"fb081f1b-a184-4656-a834-240605f9693b\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") "
Oct 31 14:02:14.988630 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.988524    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-workload-socket\") pod \"fb081f1b-a184-4656-a834-240605f9693b\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") "
Oct 31 14:02:14.988630 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.988612    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-istio-data\") pod \"fb081f1b-a184-4656-a834-240605f9693b\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") "
Oct 31 14:02:14.988869 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.988686    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb081f1b-a184-4656-a834-240605f9693b-istio-token\" (UniqueName: \"kubernetes.io/projected/fb081f1b-a184-4656-a834-240605f9693b-istio-token\") pod \"fb081f1b-a184-4656-a834-240605f9693b\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") "
Oct 31 14:02:14.988869 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.988706    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-credential-socket" pod "fb081f1b-a184-4656-a834-240605f9693b" (UID: "fb081f1b-a184-4656-a834-240605f9693b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:14.988869 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.988719    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb081f1b-a184-4656-a834-240605f9693b-kube-api-access-ljmk5\" (UniqueName: \"kubernetes.io/projected/fb081f1b-a184-4656-a834-240605f9693b-kube-api-access-ljmk5\") pod \"fb081f1b-a184-4656-a834-240605f9693b\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") "
Oct 31 14:02:14.988869 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.988803    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-workload-certs\") pod \"fb081f1b-a184-4656-a834-240605f9693b\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") "
Oct 31 14:02:14.988869 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.988826    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb081f1b-a184-4656-a834-240605f9693b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb081f1b-a184-4656-a834-240605f9693b-istio-podinfo\") pod \"fb081f1b-a184-4656-a834-240605f9693b\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") "
Oct 31 14:02:14.990257 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.988870    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-istio-envoy\") pod \"fb081f1b-a184-4656-a834-240605f9693b\" (UID: \"fb081f1b-a184-4656-a834-240605f9693b\") "
Oct 31 14:02:14.990257 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.989434    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:14.990257 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.989777    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb081f1b-a184-4656-a834-240605f9693b-istiod-ca-cert" pod "fb081f1b-a184-4656-a834-240605f9693b" (UID: "fb081f1b-a184-4656-a834-240605f9693b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:14.990257 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.990164    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-istio-data" pod "fb081f1b-a184-4656-a834-240605f9693b" (UID: "fb081f1b-a184-4656-a834-240605f9693b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:14.990257 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.990122    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-workload-certs" pod "fb081f1b-a184-4656-a834-240605f9693b" (UID: "fb081f1b-a184-4656-a834-240605f9693b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:14.990601 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.990580    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-workload-socket" pod "fb081f1b-a184-4656-a834-240605f9693b" (UID: "fb081f1b-a184-4656-a834-240605f9693b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:14.993803 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.993776    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb081f1b-a184-4656-a834-240605f9693b-kube-api-access-ljmk5" pod "fb081f1b-a184-4656-a834-240605f9693b" (UID: "fb081f1b-a184-4656-a834-240605f9693b"). InnerVolumeSpecName "kube-api-access-ljmk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:14.993803 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.993791    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-istio-envoy" pod "fb081f1b-a184-4656-a834-240605f9693b" (UID: "fb081f1b-a184-4656-a834-240605f9693b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:14.993961 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.993807    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb081f1b-a184-4656-a834-240605f9693b-istio-podinfo" pod "fb081f1b-a184-4656-a834-240605f9693b" (UID: "fb081f1b-a184-4656-a834-240605f9693b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:02:14.993961 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:14.993931    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb081f1b-a184-4656-a834-240605f9693b-istio-token" pod "fb081f1b-a184-4656-a834-240605f9693b" (UID: "fb081f1b-a184-4656-a834-240605f9693b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:15.090172 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:15.090126    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb081f1b-a184-4656-a834-240605f9693b-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:15.090172 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:15.090163    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljmk5\" (UniqueName: \"kubernetes.io/projected/fb081f1b-a184-4656-a834-240605f9693b-kube-api-access-ljmk5\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:15.090172 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:15.090174    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:15.090172 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:15.090183    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb081f1b-a184-4656-a834-240605f9693b-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:15.090557 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:15.090192    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:15.090557 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:15.090200    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb081f1b-a184-4656-a834-240605f9693b-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:15.090557 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:15.090209    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:15.090557 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:15.090218    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb081f1b-a184-4656-a834-240605f9693b-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:15.384281 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:15.384248    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"
Oct 31 14:02:15.384504 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:15.384279    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9" event={"ID":"fb081f1b-a184-4656-a834-240605f9693b","Type":"ContainerDied","Data":"f2c27fb5d20d340d95d4a879fea238d17662d4eab745e68d0c87c0689bf49291"}
Oct 31 14:02:15.384504 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:15.384420    4660 scope.go:122] "RemoveContainer" containerID="0c543a4a65bbfb2b89063ee6f745227e6863eadf7b0b6fc716a7f44ff86f7558"
Oct 31 14:02:15.411780 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:15.411740    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"]
Oct 31 14:02:15.417994 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:15.417960    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-hv9c9"]
Oct 31 14:02:16.531014 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:16.530975    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db28b27-5dcf-4468-8ebc-951e8f52613f" path="/var/lib/kubelet/pods/5db28b27-5dcf-4468-8ebc-951e8f52613f/volumes"
Oct 31 14:02:16.531727 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:16.531706    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb081f1b-a184-4656-a834-240605f9693b" path="/var/lib/kubelet/pods/fb081f1b-a184-4656-a834-240605f9693b/volumes"
Oct 31 14:02:17.398966 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:17.398926    4660 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:02:17.399149 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:17.399008    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:02:19.754369 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.754341    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 14:02:19.872510 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.872471    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-workload-socket\") pod \"aa19833b-88b6-453e-81c8-4ce97c30669e\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") "
Oct 31 14:02:19.872887 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.872869    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-workload-certs\") pod \"aa19833b-88b6-453e-81c8-4ce97c30669e\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") "
Oct 31 14:02:19.873699 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.873039    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-data\") pod \"aa19833b-88b6-453e-81c8-4ce97c30669e\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") "
Oct 31 14:02:19.873864 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.872948    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-workload-socket" pod "aa19833b-88b6-453e-81c8-4ce97c30669e" (UID: "aa19833b-88b6-453e-81c8-4ce97c30669e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:19.873864 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.873166    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-workload-certs" pod "aa19833b-88b6-453e-81c8-4ce97c30669e" (UID: "aa19833b-88b6-453e-81c8-4ce97c30669e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:19.873864 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.873418    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-data" pod "aa19833b-88b6-453e-81c8-4ce97c30669e" (UID: "aa19833b-88b6-453e-81c8-4ce97c30669e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:19.874064 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.873874    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa19833b-88b6-453e-81c8-4ce97c30669e-kube-api-access-hpcq9\" (UniqueName: \"kubernetes.io/projected/aa19833b-88b6-453e-81c8-4ce97c30669e-kube-api-access-hpcq9\") pod \"aa19833b-88b6-453e-81c8-4ce97c30669e\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") "
Oct 31 14:02:19.874064 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.874013    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-podinfo\") pod \"aa19833b-88b6-453e-81c8-4ce97c30669e\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") "
Oct 31 14:02:19.874064 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.874051    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa19833b-88b6-453e-81c8-4ce97c30669e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa19833b-88b6-453e-81c8-4ce97c30669e-istiod-ca-cert\") pod \"aa19833b-88b6-453e-81c8-4ce97c30669e\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") "
Oct 31 14:02:19.874236 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.874091    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-credential-socket\") pod \"aa19833b-88b6-453e-81c8-4ce97c30669e\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") "
Oct 31 14:02:19.874236 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.874120    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-token\" (UniqueName: \"kubernetes.io/projected/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-token\") pod \"aa19833b-88b6-453e-81c8-4ce97c30669e\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") "
Oct 31 14:02:19.874236 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.874147    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-envoy\") pod \"aa19833b-88b6-453e-81c8-4ce97c30669e\" (UID: \"aa19833b-88b6-453e-81c8-4ce97c30669e\") "
Oct 31 14:02:19.874630 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.874587    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa19833b-88b6-453e-81c8-4ce97c30669e-istiod-ca-cert" pod "aa19833b-88b6-453e-81c8-4ce97c30669e" (UID: "aa19833b-88b6-453e-81c8-4ce97c30669e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:19.874806 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.874787    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa19833b-88b6-453e-81c8-4ce97c30669e-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.874880 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.874810    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.874880 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.874838    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.874880 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.874850    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.875566 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.875540    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-credential-socket" pod "aa19833b-88b6-453e-81c8-4ce97c30669e" (UID: "aa19833b-88b6-453e-81c8-4ce97c30669e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:19.881027 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.879717    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa19833b-88b6-453e-81c8-4ce97c30669e-kube-api-access-hpcq9" pod "aa19833b-88b6-453e-81c8-4ce97c30669e" (UID: "aa19833b-88b6-453e-81c8-4ce97c30669e"). InnerVolumeSpecName "kube-api-access-hpcq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:19.881027 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.879788    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-podinfo" pod "aa19833b-88b6-453e-81c8-4ce97c30669e" (UID: "aa19833b-88b6-453e-81c8-4ce97c30669e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:02:19.883217 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.883192    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-token" pod "aa19833b-88b6-453e-81c8-4ce97c30669e" (UID: "aa19833b-88b6-453e-81c8-4ce97c30669e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:19.883395 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.883219    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-envoy" pod "aa19833b-88b6-453e-81c8-4ce97c30669e" (UID: "aa19833b-88b6-453e-81c8-4ce97c30669e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:19.976278 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.976234    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpcq9\" (UniqueName: \"kubernetes.io/projected/aa19833b-88b6-453e-81c8-4ce97c30669e-kube-api-access-hpcq9\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.976278 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.976275    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.976278 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.976304    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.976278 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.976319    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.976694 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:19.976332    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa19833b-88b6-453e-81c8-4ce97c30669e-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:20.431086 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:20.431038    4660 generic.go:356] "Generic (PLEG): container finished" podID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerID="5857f4768517a5769b30be330ccf13c59e589f0860b7e3bcfdd977e9da83330d" exitCode=0
Oct 31 14:02:20.431337 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:20.431120    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"
Oct 31 14:02:20.431450 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:20.431121    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" event={"ID":"aa19833b-88b6-453e-81c8-4ce97c30669e","Type":"ContainerDied","Data":"5857f4768517a5769b30be330ccf13c59e589f0860b7e3bcfdd977e9da83330d"}
Oct 31 14:02:20.431612 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:20.431526    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs" event={"ID":"aa19833b-88b6-453e-81c8-4ce97c30669e","Type":"ContainerDied","Data":"2f1184e530498da5eb479f2f861bc0834c50812fd5b5d22ea90d7e0e15c01688"}
Oct 31 14:02:20.431612 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:20.431565    4660 scope.go:122] "RemoveContainer" containerID="5857f4768517a5769b30be330ccf13c59e589f0860b7e3bcfdd977e9da83330d"
Oct 31 14:02:20.445199 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:20.445172    4660 scope.go:122] "RemoveContainer" containerID="5857f4768517a5769b30be330ccf13c59e589f0860b7e3bcfdd977e9da83330d"
Oct 31 14:02:20.445599 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:02:20.445578    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5857f4768517a5769b30be330ccf13c59e589f0860b7e3bcfdd977e9da83330d\": container with ID starting with 5857f4768517a5769b30be330ccf13c59e589f0860b7e3bcfdd977e9da83330d not found: ID does not exist" containerID="5857f4768517a5769b30be330ccf13c59e589f0860b7e3bcfdd977e9da83330d"
Oct 31 14:02:20.445679 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:20.445610    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5857f4768517a5769b30be330ccf13c59e589f0860b7e3bcfdd977e9da83330d"} err="failed to get container status \"5857f4768517a5769b30be330ccf13c59e589f0860b7e3bcfdd977e9da83330d\": rpc error: code = NotFound desc = could not find container \"5857f4768517a5769b30be330ccf13c59e589f0860b7e3bcfdd977e9da83330d\": container with ID starting with 5857f4768517a5769b30be330ccf13c59e589f0860b7e3bcfdd977e9da83330d not found: ID does not exist"
Oct 31 14:02:20.464050 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:20.464019    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"]
Oct 31 14:02:20.477462 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:20.467887    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-w7bgs"]
Oct 31 14:02:20.527008 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:20.526972    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" path="/var/lib/kubelet/pods/aa19833b-88b6-453e-81c8-4ce97c30669e/volumes"
Oct 31 14:02:20.748387 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:20.745993    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"]
Oct 31 14:02:20.748387 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:20.746264    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j" podUID="5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" containerName="istio-proxy" containerID="cri-o://2ce6dad9f56edee1449510a709103a7a5d078b7a9c18279f6218974a9804c3da" gracePeriod=30
Oct 31 14:02:20.795955 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:02:20.795909    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:02:26.452109 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.452078    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 14:02:26.484308 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.484253    4660 generic.go:356] "Generic (PLEG): container finished" podID="5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" containerID="2ce6dad9f56edee1449510a709103a7a5d078b7a9c18279f6218974a9804c3da" exitCode=0
Oct 31 14:02:26.484521 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.484326    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"
Oct 31 14:02:26.484521 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.484329    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j" event={"ID":"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3","Type":"ContainerDied","Data":"2ce6dad9f56edee1449510a709103a7a5d078b7a9c18279f6218974a9804c3da"}
Oct 31 14:02:26.484521 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.484412    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j" event={"ID":"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3","Type":"ContainerDied","Data":"845ccf7eb89c60a0bf3ebf4af703be265b7a1419a4965ce4a3fe6080acff4c56"}
Oct 31 14:02:26.484521 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.484451    4660 scope.go:122] "RemoveContainer" containerID="2ce6dad9f56edee1449510a709103a7a5d078b7a9c18279f6218974a9804c3da"
Oct 31 14:02:26.501032 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.500998    4660 scope.go:122] "RemoveContainer" containerID="2ce6dad9f56edee1449510a709103a7a5d078b7a9c18279f6218974a9804c3da"
Oct 31 14:02:26.501470 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:02:26.501443    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ce6dad9f56edee1449510a709103a7a5d078b7a9c18279f6218974a9804c3da\": container with ID starting with 2ce6dad9f56edee1449510a709103a7a5d078b7a9c18279f6218974a9804c3da not found: ID does not exist" containerID="2ce6dad9f56edee1449510a709103a7a5d078b7a9c18279f6218974a9804c3da"
Oct 31 14:02:26.501576 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.501483    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce6dad9f56edee1449510a709103a7a5d078b7a9c18279f6218974a9804c3da"} err="failed to get container status \"2ce6dad9f56edee1449510a709103a7a5d078b7a9c18279f6218974a9804c3da\": rpc error: code = NotFound desc = could not find container \"2ce6dad9f56edee1449510a709103a7a5d078b7a9c18279f6218974a9804c3da\": container with ID starting with 2ce6dad9f56edee1449510a709103a7a5d078b7a9c18279f6218974a9804c3da not found: ID does not exist"
Oct 31 14:02:26.591526 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.591446    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-kube-api-access-q87f4\" (UniqueName: \"kubernetes.io/projected/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-kube-api-access-q87f4\") pod \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") "
Oct 31 14:02:26.591526 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.591487    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-envoy\") pod \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") "
Oct 31 14:02:26.591526 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.591527    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-podinfo\") pod \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") "
Oct 31 14:02:26.591836 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.591560    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-workload-socket\") pod \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") "
Oct 31 14:02:26.591836 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.591593    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istiod-ca-cert\") pod \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") "
Oct 31 14:02:26.591836 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.591622    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-workload-certs\") pod \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") "
Oct 31 14:02:26.591836 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.591652    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-data\") pod \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") "
Oct 31 14:02:26.591836 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.591724    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-credential-socket\") pod \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") "
Oct 31 14:02:26.591836 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.591756    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-token\" (UniqueName: \"kubernetes.io/projected/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-token\") pod \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\" (UID: \"5549d7d1-08ba-419c-8cf1-9c2b29ceaec3\") "
Oct 31 14:02:26.592196 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.591900    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-workload-socket" pod "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" (UID: "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:26.592196 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.592131    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-workload-certs" pod "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" (UID: "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:26.592196 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.592147    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-data" pod "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" (UID: "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:26.592473 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.592367    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.592473 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.592392    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.592473 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.592408    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.592654 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.592490    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istiod-ca-cert" pod "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" (UID: "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:26.592654 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.592587    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-credential-socket" pod "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" (UID: "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:26.594482 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.594456    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-envoy" pod "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" (UID: "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:26.594618 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.594551    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-token" pod "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" (UID: "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:26.594692 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.594671    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-kube-api-access-q87f4" pod "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" (UID: "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3"). InnerVolumeSpecName "kube-api-access-q87f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:26.595416 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.595394    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-podinfo" pod "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" (UID: "5549d7d1-08ba-419c-8cf1-9c2b29ceaec3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:02:26.697135 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.697095    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.697135 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.697133    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.697135 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.697149    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q87f4\" (UniqueName: \"kubernetes.io/projected/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-kube-api-access-q87f4\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.697527 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.697166    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.697527 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.697179    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.697527 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.697192    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.816164 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.816099    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"]
Oct 31 14:02:26.819277 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:26.819247    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-kq25j"]
Oct 31 14:02:28.527893 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:28.527856    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" path="/var/lib/kubelet/pods/5549d7d1-08ba-419c-8cf1-9c2b29ceaec3/volumes"
Oct 31 14:02:29.077785 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.077735    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" containerID="cri-o://5d32dc720c70379446093b674ea661e2135d4e36c673e8c11926917141fd5e2f" gracePeriod=45
Oct 31 14:02:29.082147 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.082110    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": EOF"
Oct 31 14:02:29.358439 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.358405    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 14:02:29.509908 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.509873    4660 generic.go:356] "Generic (PLEG): container finished" podID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerID="5d32dc720c70379446093b674ea661e2135d4e36c673e8c11926917141fd5e2f" exitCode=0
Oct 31 14:02:29.510084 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.509919    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"
Oct 31 14:02:29.510084 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.509955    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" event={"ID":"28afbd7d-8b18-44e6-8023-8a1019197ceb","Type":"ContainerDied","Data":"5d32dc720c70379446093b674ea661e2135d4e36c673e8c11926917141fd5e2f"}
Oct 31 14:02:29.510084 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.510001    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w" event={"ID":"28afbd7d-8b18-44e6-8023-8a1019197ceb","Type":"ContainerDied","Data":"8f28a4470559bbe0e99f395f85272c8d84d59fc4ef15481a2df9536886cce8ae"}
Oct 31 14:02:29.510084 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.510023    4660 scope.go:122] "RemoveContainer" containerID="5d32dc720c70379446093b674ea661e2135d4e36c673e8c11926917141fd5e2f"
Oct 31 14:02:29.520340 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.520319    4660 scope.go:122] "RemoveContainer" containerID="5d32dc720c70379446093b674ea661e2135d4e36c673e8c11926917141fd5e2f"
Oct 31 14:02:29.520685 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:02:29.520661    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d32dc720c70379446093b674ea661e2135d4e36c673e8c11926917141fd5e2f\": container with ID starting with 5d32dc720c70379446093b674ea661e2135d4e36c673e8c11926917141fd5e2f not found: ID does not exist" containerID="5d32dc720c70379446093b674ea661e2135d4e36c673e8c11926917141fd5e2f"
Oct 31 14:02:29.520761 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.520699    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d32dc720c70379446093b674ea661e2135d4e36c673e8c11926917141fd5e2f"} err="failed to get container status \"5d32dc720c70379446093b674ea661e2135d4e36c673e8c11926917141fd5e2f\": rpc error: code = NotFound desc = could not find container \"5d32dc720c70379446093b674ea661e2135d4e36c673e8c11926917141fd5e2f\": container with ID starting with 5d32dc720c70379446093b674ea661e2135d4e36c673e8c11926917141fd5e2f not found: ID does not exist"
Oct 31 14:02:29.532164 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.532135    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-dshm\") pod \"28afbd7d-8b18-44e6-8023-8a1019197ceb\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") "
Oct 31 14:02:29.532595 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.532203    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28afbd7d-8b18-44e6-8023-8a1019197ceb-tls-certs\" (UniqueName: \"kubernetes.io/secret/28afbd7d-8b18-44e6-8023-8a1019197ceb-tls-certs\") pod \"28afbd7d-8b18-44e6-8023-8a1019197ceb\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") "
Oct 31 14:02:29.532595 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.532266    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-tmp-dir\") pod \"28afbd7d-8b18-44e6-8023-8a1019197ceb\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") "
Oct 31 14:02:29.532595 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.532320    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-model-cache\") pod \"28afbd7d-8b18-44e6-8023-8a1019197ceb\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") "
Oct 31 14:02:29.532595 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.532435    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-home\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-home\") pod \"28afbd7d-8b18-44e6-8023-8a1019197ceb\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") "
Oct 31 14:02:29.532595 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.532587    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28afbd7d-8b18-44e6-8023-8a1019197ceb-kube-api-access-bb4qj\" (UniqueName: \"kubernetes.io/projected/28afbd7d-8b18-44e6-8023-8a1019197ceb-kube-api-access-bb4qj\") pod \"28afbd7d-8b18-44e6-8023-8a1019197ceb\" (UID: \"28afbd7d-8b18-44e6-8023-8a1019197ceb\") "
Oct 31 14:02:29.532934 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.532624    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-model-cache" pod "28afbd7d-8b18-44e6-8023-8a1019197ceb" (UID: "28afbd7d-8b18-44e6-8023-8a1019197ceb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:29.532934 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.532790    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-home" pod "28afbd7d-8b18-44e6-8023-8a1019197ceb" (UID: "28afbd7d-8b18-44e6-8023-8a1019197ceb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:29.533101 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.533075    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:29.533244 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.533136    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:29.533244 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.533143    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-tmp-dir" pod "28afbd7d-8b18-44e6-8023-8a1019197ceb" (UID: "28afbd7d-8b18-44e6-8023-8a1019197ceb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:29.534737 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.534711    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-dshm" pod "28afbd7d-8b18-44e6-8023-8a1019197ceb" (UID: "28afbd7d-8b18-44e6-8023-8a1019197ceb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:29.534945 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.534924    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28afbd7d-8b18-44e6-8023-8a1019197ceb-tls-certs" pod "28afbd7d-8b18-44e6-8023-8a1019197ceb" (UID: "28afbd7d-8b18-44e6-8023-8a1019197ceb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:29.535044 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.535001    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28afbd7d-8b18-44e6-8023-8a1019197ceb-kube-api-access-bb4qj" pod "28afbd7d-8b18-44e6-8023-8a1019197ceb" (UID: "28afbd7d-8b18-44e6-8023-8a1019197ceb"). InnerVolumeSpecName "kube-api-access-bb4qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:29.634111 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.634072    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb4qj\" (UniqueName: \"kubernetes.io/projected/28afbd7d-8b18-44e6-8023-8a1019197ceb-kube-api-access-bb4qj\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:29.634111 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.634106    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:29.634111 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.634116    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28afbd7d-8b18-44e6-8023-8a1019197ceb-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:29.634455 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.634125    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28afbd7d-8b18-44e6-8023-8a1019197ceb-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:29.844641 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.844598    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"]
Oct 31 14:02:29.849356 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:29.849325    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/distinct-w5-6784d6-kserve-9747d98cc-8sh4w"]
Oct 31 14:02:30.527933 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:30.527901    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" path="/var/lib/kubelet/pods/28afbd7d-8b18-44e6-8023-8a1019197ceb/volumes"
Oct 31 14:02:31.440778 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:31.438995    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"]
Oct 31 14:02:31.621082 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:31.621030    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"]
Oct 31 14:02:31.627731 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:31.626777    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv" podUID="ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" containerName="istio-proxy" containerID="cri-o://d7dd1a939bdbc9174d69028994cb7d7f9344bc38eecef91d05b56901741fba03" gracePeriod=30
Oct 31 14:02:32.424128 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:32.424086    4660 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:02:32.424366 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:32.424147    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv" podUID="ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:02:36.440588 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.439760    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw" podUID="84174b11-1866-4f83-ad10-84a6288b53dd" containerName="payload-processing" containerID="cri-o://9d5ed341b35fee7a96a7b4bb55cab9e6e67ddc86fcd98e941140e6a8690ae92e" gracePeriod=25
Oct 31 14:02:36.913407 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.913322    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 14:02:36.935344 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.935282    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-envoy\") pod \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") "
Oct 31 14:02:36.935550 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.935385    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-data\") pod \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") "
Oct 31 14:02:36.935550 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.935442    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-workload-socket\") pod \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") "
Oct 31 14:02:36.935550 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.935478    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-kube-api-access-4wgl7\" (UniqueName: \"kubernetes.io/projected/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-kube-api-access-4wgl7\") pod \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") "
Oct 31 14:02:36.935550 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.935529    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-podinfo\") pod \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") "
Oct 31 14:02:36.935782 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.935570    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istiod-ca-cert\") pod \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") "
Oct 31 14:02:36.935782 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.935626    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-workload-certs\") pod \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") "
Oct 31 14:02:36.935782 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.935683    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-credential-socket\") pod \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") "
Oct 31 14:02:36.935782 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.935712    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-data" pod "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" (UID: "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:36.936064 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.935811    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-token\" (UniqueName: \"kubernetes.io/projected/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-token\") pod \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\" (UID: \"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2\") "
Oct 31 14:02:36.936064 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.935958    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-workload-socket" pod "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" (UID: "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:36.936190 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.936166    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-credential-socket" pod "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" (UID: "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:36.936246 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.936229    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:36.936345 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.936252    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:36.936415 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.936392    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-workload-certs" pod "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" (UID: "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:36.936541 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.936513    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istiod-ca-cert" pod "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" (UID: "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:36.939048 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.939003    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-token" pod "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" (UID: "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:36.939048 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.939025    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-kube-api-access-4wgl7" pod "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" (UID: "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2"). InnerVolumeSpecName "kube-api-access-4wgl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:36.939386 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.939256    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-envoy" pod "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" (UID: "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:36.939386 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:36.939353    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-podinfo" pod "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" (UID: "ed888973-2d46-4359-8d0e-dd2e7f6fc0b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:02:37.037257 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.037218    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.037257 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.037253    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.037257 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.037262    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.037528 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.037271    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.037528 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.037281    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wgl7\" (UniqueName: \"kubernetes.io/projected/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-kube-api-access-4wgl7\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.037528 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.037303    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.037528 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.037313    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.150884 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.150838    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"]
Oct 31 14:02:37.343371 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.340767    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"]
Oct 31 14:02:37.353318 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.348326    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq" podUID="04449154-f556-4576-b8c1-1a4ec39c960a" containerName="maas-api" containerID="cri-o://00cc6457f30436dfd385a27d6e736a3c5bfbe88bbf492ea47d01555da8276611" gracePeriod=30
Oct 31 14:02:37.684882 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.674220    4660 generic.go:356] "Generic (PLEG): container finished" podID="04449154-f556-4576-b8c1-1a4ec39c960a" containerID="00cc6457f30436dfd385a27d6e736a3c5bfbe88bbf492ea47d01555da8276611" exitCode=0
Oct 31 14:02:37.684882 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.674427    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq" event={"ID":"04449154-f556-4576-b8c1-1a4ec39c960a","Type":"ContainerDied","Data":"00cc6457f30436dfd385a27d6e736a3c5bfbe88bbf492ea47d01555da8276611"}
Oct 31 14:02:37.684882 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.676310    4660 generic.go:356] "Generic (PLEG): container finished" podID="ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" containerID="d7dd1a939bdbc9174d69028994cb7d7f9344bc38eecef91d05b56901741fba03" exitCode=0
Oct 31 14:02:37.684882 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.676394    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv" event={"ID":"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2","Type":"ContainerDied","Data":"d7dd1a939bdbc9174d69028994cb7d7f9344bc38eecef91d05b56901741fba03"}
Oct 31 14:02:37.684882 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.676434    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv" event={"ID":"ed888973-2d46-4359-8d0e-dd2e7f6fc0b2","Type":"ContainerDied","Data":"22d1185cfe69de9d657be4d35bd8d4ceb2ff91b38ff58ee211497eac61dbc4b7"}
Oct 31 14:02:37.684882 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.676460    4660 scope.go:122] "RemoveContainer" containerID="d7dd1a939bdbc9174d69028994cb7d7f9344bc38eecef91d05b56901741fba03"
Oct 31 14:02:37.684882 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.676619    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"
Oct 31 14:02:37.705987 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.705924    4660 scope.go:122] "RemoveContainer" containerID="d7dd1a939bdbc9174d69028994cb7d7f9344bc38eecef91d05b56901741fba03"
Oct 31 14:02:37.706538 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:02:37.706399    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7dd1a939bdbc9174d69028994cb7d7f9344bc38eecef91d05b56901741fba03\": container with ID starting with d7dd1a939bdbc9174d69028994cb7d7f9344bc38eecef91d05b56901741fba03 not found: ID does not exist" containerID="d7dd1a939bdbc9174d69028994cb7d7f9344bc38eecef91d05b56901741fba03"
Oct 31 14:02:37.706538 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.706448    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7dd1a939bdbc9174d69028994cb7d7f9344bc38eecef91d05b56901741fba03"} err="failed to get container status \"d7dd1a939bdbc9174d69028994cb7d7f9344bc38eecef91d05b56901741fba03\": rpc error: code = NotFound desc = could not find container \"d7dd1a939bdbc9174d69028994cb7d7f9344bc38eecef91d05b56901741fba03\": container with ID starting with d7dd1a939bdbc9174d69028994cb7d7f9344bc38eecef91d05b56901741fba03 not found: ID does not exist"
Oct 31 14:02:37.727085 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.722339    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"]
Oct 31 14:02:37.727269 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.727214    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-flxpv"]
Oct 31 14:02:37.798027 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.797984    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 14:02:37.885316 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.880486    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04449154-f556-4576-b8c1-1a4ec39c960a-kube-api-access-x5ht4\" (UniqueName: \"kubernetes.io/projected/04449154-f556-4576-b8c1-1a4ec39c960a-kube-api-access-x5ht4\") pod \"04449154-f556-4576-b8c1-1a4ec39c960a\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") "
Oct 31 14:02:37.885316 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.880559    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04449154-f556-4576-b8c1-1a4ec39c960a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04449154-f556-4576-b8c1-1a4ec39c960a-openshift-service-ca\") pod \"04449154-f556-4576-b8c1-1a4ec39c960a\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") "
Oct 31 14:02:37.885316 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.880670    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-maas-api-tls\") pod \"04449154-f556-4576-b8c1-1a4ec39c960a\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") "
Oct 31 14:02:37.885316 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.880771    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-metrics-certs\") pod \"04449154-f556-4576-b8c1-1a4ec39c960a\" (UID: \"04449154-f556-4576-b8c1-1a4ec39c960a\") "
Oct 31 14:02:37.885316 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.882381    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04449154-f556-4576-b8c1-1a4ec39c960a-openshift-service-ca" pod "04449154-f556-4576-b8c1-1a4ec39c960a" (UID: "04449154-f556-4576-b8c1-1a4ec39c960a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:37.892695 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.892640    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-metrics-certs" pod "04449154-f556-4576-b8c1-1a4ec39c960a" (UID: "04449154-f556-4576-b8c1-1a4ec39c960a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:37.894843 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.894809    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-maas-api-tls" pod "04449154-f556-4576-b8c1-1a4ec39c960a" (UID: "04449154-f556-4576-b8c1-1a4ec39c960a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:37.895045 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.895021    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04449154-f556-4576-b8c1-1a4ec39c960a-kube-api-access-x5ht4" pod "04449154-f556-4576-b8c1-1a4ec39c960a" (UID: "04449154-f556-4576-b8c1-1a4ec39c960a"). InnerVolumeSpecName "kube-api-access-x5ht4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:37.982207 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.982116    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.982207 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.982157    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5ht4\" (UniqueName: \"kubernetes.io/projected/04449154-f556-4576-b8c1-1a4ec39c960a-kube-api-access-x5ht4\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.982207 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.982177    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04449154-f556-4576-b8c1-1a4ec39c960a-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.982207 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:37.982193    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04449154-f556-4576-b8c1-1a4ec39c960a-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:38.530475 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:38.530433    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" path="/var/lib/kubelet/pods/ed888973-2d46-4359-8d0e-dd2e7f6fc0b2/volumes"
Oct 31 14:02:38.688089 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:38.688053    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"
Oct 31 14:02:38.688622 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:38.688052    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq" event={"ID":"04449154-f556-4576-b8c1-1a4ec39c960a","Type":"ContainerDied","Data":"068a1d8dc0a96df695be0f1385f32db6dfecf2d306f675564bb5eb1022656124"}
Oct 31 14:02:38.688622 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:38.688213    4660 scope.go:122] "RemoveContainer" containerID="00cc6457f30436dfd385a27d6e736a3c5bfbe88bbf492ea47d01555da8276611"
Oct 31 14:02:38.712714 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:38.712679    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"]
Oct 31 14:02:38.715902 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:38.715855    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6784d6-675775b557-v99vq"]
Oct 31 14:02:40.527032 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:40.526991    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04449154-f556-4576-b8c1-1a4ec39c960a" path="/var/lib/kubelet/pods/04449154-f556-4576-b8c1-1a4ec39c960a/volumes"
Oct 31 14:02:42.845553 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:42.845509    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw" podUID="84174b11-1866-4f83-ad10-84a6288b53dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:45.956358 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:45.951120    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"]
Oct 31 14:02:45.956358 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:45.951849    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j" podUID="6a36003d-cda6-447e-bb9f-95f13e32c276" containerName="istio-proxy" containerID="cri-o://0ddafa001e3e28c27e8f86d931ed37ee0e22eed1ea065d4bbfbbe17bdcb5afaf" gracePeriod=30
Oct 31 14:02:50.737274 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:02:50.737209    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:02:51.227396 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.227349    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"]
Oct 31 14:02:51.227644 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.227618    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz" podUID="3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e" containerName="maas-api" containerID="cri-o://5368d6c18711b7d9e12226daa599ccd1bf81b1072e6e83897f2abd82929719af" gracePeriod=30
Oct 31 14:02:51.383775 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.383746    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 14:02:51.513628 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.513502    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-workload-certs" pod "6a36003d-cda6-447e-bb9f-95f13e32c276" (UID: "6a36003d-cda6-447e-bb9f-95f13e32c276"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:51.513930 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.513879    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-workload-certs\") pod \"6a36003d-cda6-447e-bb9f-95f13e32c276\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") "
Oct 31 14:02:51.514457 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.514410    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-workload-socket" pod "6a36003d-cda6-447e-bb9f-95f13e32c276" (UID: "6a36003d-cda6-447e-bb9f-95f13e32c276"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:51.514628 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.514611    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-workload-socket\") pod \"6a36003d-cda6-447e-bb9f-95f13e32c276\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") "
Oct 31 14:02:51.515183 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.514790    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-envoy\") pod \"6a36003d-cda6-447e-bb9f-95f13e32c276\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") "
Oct 31 14:02:51.515449 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.515429    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-podinfo\") pod \"6a36003d-cda6-447e-bb9f-95f13e32c276\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") "
Oct 31 14:02:51.515987 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.515969    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a36003d-cda6-447e-bb9f-95f13e32c276-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a36003d-cda6-447e-bb9f-95f13e32c276-istiod-ca-cert\") pod \"6a36003d-cda6-447e-bb9f-95f13e32c276\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") "
Oct 31 14:02:51.516174 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.516148    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-data\") pod \"6a36003d-cda6-447e-bb9f-95f13e32c276\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") "
Oct 31 14:02:51.516351 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.516328    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-token\" (UniqueName: \"kubernetes.io/projected/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-token\") pod \"6a36003d-cda6-447e-bb9f-95f13e32c276\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") "
Oct 31 14:02:51.516601 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.516581    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a36003d-cda6-447e-bb9f-95f13e32c276-kube-api-access-rrlg2\" (UniqueName: \"kubernetes.io/projected/6a36003d-cda6-447e-bb9f-95f13e32c276-kube-api-access-rrlg2\") pod \"6a36003d-cda6-447e-bb9f-95f13e32c276\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") "
Oct 31 14:02:51.516806 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.516779    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-credential-socket\") pod \"6a36003d-cda6-447e-bb9f-95f13e32c276\" (UID: \"6a36003d-cda6-447e-bb9f-95f13e32c276\") "
Oct 31 14:02:51.517735 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.517715    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:51.517890 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.517875    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:51.518261 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.518243    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-credential-socket" pod "6a36003d-cda6-447e-bb9f-95f13e32c276" (UID: "6a36003d-cda6-447e-bb9f-95f13e32c276"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:51.518787 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.518756    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a36003d-cda6-447e-bb9f-95f13e32c276-istiod-ca-cert" pod "6a36003d-cda6-447e-bb9f-95f13e32c276" (UID: "6a36003d-cda6-447e-bb9f-95f13e32c276"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:51.519807 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.519783    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-data" pod "6a36003d-cda6-447e-bb9f-95f13e32c276" (UID: "6a36003d-cda6-447e-bb9f-95f13e32c276"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:51.525740 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.525696    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a36003d-cda6-447e-bb9f-95f13e32c276-kube-api-access-rrlg2" pod "6a36003d-cda6-447e-bb9f-95f13e32c276" (UID: "6a36003d-cda6-447e-bb9f-95f13e32c276"). InnerVolumeSpecName "kube-api-access-rrlg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:51.532158 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.532102    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-envoy" pod "6a36003d-cda6-447e-bb9f-95f13e32c276" (UID: "6a36003d-cda6-447e-bb9f-95f13e32c276"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:51.532666 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.532622    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-token" pod "6a36003d-cda6-447e-bb9f-95f13e32c276" (UID: "6a36003d-cda6-447e-bb9f-95f13e32c276"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:51.533514 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.533474    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-podinfo" pod "6a36003d-cda6-447e-bb9f-95f13e32c276" (UID: "6a36003d-cda6-447e-bb9f-95f13e32c276"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:02:51.620270 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.620229    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:51.620270 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.620268    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:51.620749 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.620283    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrlg2\" (UniqueName: \"kubernetes.io/projected/6a36003d-cda6-447e-bb9f-95f13e32c276-kube-api-access-rrlg2\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:51.620749 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.620315    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:51.620749 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.620330    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:51.620749 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.620345    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a36003d-cda6-447e-bb9f-95f13e32c276-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:51.620749 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.620359    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a36003d-cda6-447e-bb9f-95f13e32c276-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:51.688004 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.687964    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 14:02:51.790507 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.790404    4660 generic.go:356] "Generic (PLEG): container finished" podID="3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e" containerID="5368d6c18711b7d9e12226daa599ccd1bf81b1072e6e83897f2abd82929719af" exitCode=0
Oct 31 14:02:51.790507 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.790442    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz" event={"ID":"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e","Type":"ContainerDied","Data":"5368d6c18711b7d9e12226daa599ccd1bf81b1072e6e83897f2abd82929719af"}
Oct 31 14:02:51.790507 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.790498    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"
Oct 31 14:02:51.791061 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.790534    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz" event={"ID":"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e","Type":"ContainerDied","Data":"7972cc00e6429124f86078fcf411100eed836c8fc6640b764ceb8a7961f31a74"}
Oct 31 14:02:51.791061 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.790559    4660 scope.go:122] "RemoveContainer" containerID="5368d6c18711b7d9e12226daa599ccd1bf81b1072e6e83897f2abd82929719af"
Oct 31 14:02:51.792466 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.792442    4660 generic.go:356] "Generic (PLEG): container finished" podID="6a36003d-cda6-447e-bb9f-95f13e32c276" containerID="0ddafa001e3e28c27e8f86d931ed37ee0e22eed1ea065d4bbfbbe17bdcb5afaf" exitCode=0
Oct 31 14:02:51.792612 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.792478    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j" event={"ID":"6a36003d-cda6-447e-bb9f-95f13e32c276","Type":"ContainerDied","Data":"0ddafa001e3e28c27e8f86d931ed37ee0e22eed1ea065d4bbfbbe17bdcb5afaf"}
Oct 31 14:02:51.792612 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.792522    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"
Oct 31 14:02:51.792612 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.792535    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j" event={"ID":"6a36003d-cda6-447e-bb9f-95f13e32c276","Type":"ContainerDied","Data":"2748aa8f1f91707746ccd3de41fe95b2c567c8e3438ad76912e0892e965d1621"}
Oct 31 14:02:51.804817 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.804793    4660 scope.go:122] "RemoveContainer" containerID="5368d6c18711b7d9e12226daa599ccd1bf81b1072e6e83897f2abd82929719af"
Oct 31 14:02:51.805243 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:02:51.805214    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5368d6c18711b7d9e12226daa599ccd1bf81b1072e6e83897f2abd82929719af\": container with ID starting with 5368d6c18711b7d9e12226daa599ccd1bf81b1072e6e83897f2abd82929719af not found: ID does not exist" containerID="5368d6c18711b7d9e12226daa599ccd1bf81b1072e6e83897f2abd82929719af"
Oct 31 14:02:51.805454 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.805418    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5368d6c18711b7d9e12226daa599ccd1bf81b1072e6e83897f2abd82929719af"} err="failed to get container status \"5368d6c18711b7d9e12226daa599ccd1bf81b1072e6e83897f2abd82929719af\": rpc error: code = NotFound desc = could not find container \"5368d6c18711b7d9e12226daa599ccd1bf81b1072e6e83897f2abd82929719af\": container with ID starting with 5368d6c18711b7d9e12226daa599ccd1bf81b1072e6e83897f2abd82929719af not found: ID does not exist"
Oct 31 14:02:51.805454 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.805455    4660 scope.go:122] "RemoveContainer" containerID="0ddafa001e3e28c27e8f86d931ed37ee0e22eed1ea065d4bbfbbe17bdcb5afaf"
Oct 31 14:02:51.819617 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.819586    4660 scope.go:122] "RemoveContainer" containerID="0ddafa001e3e28c27e8f86d931ed37ee0e22eed1ea065d4bbfbbe17bdcb5afaf"
Oct 31 14:02:51.819988 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:02:51.819962    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ddafa001e3e28c27e8f86d931ed37ee0e22eed1ea065d4bbfbbe17bdcb5afaf\": container with ID starting with 0ddafa001e3e28c27e8f86d931ed37ee0e22eed1ea065d4bbfbbe17bdcb5afaf not found: ID does not exist" containerID="0ddafa001e3e28c27e8f86d931ed37ee0e22eed1ea065d4bbfbbe17bdcb5afaf"
Oct 31 14:02:51.820083 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.819999    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ddafa001e3e28c27e8f86d931ed37ee0e22eed1ea065d4bbfbbe17bdcb5afaf"} err="failed to get container status \"0ddafa001e3e28c27e8f86d931ed37ee0e22eed1ea065d4bbfbbe17bdcb5afaf\": rpc error: code = NotFound desc = could not find container \"0ddafa001e3e28c27e8f86d931ed37ee0e22eed1ea065d4bbfbbe17bdcb5afaf\": container with ID starting with 0ddafa001e3e28c27e8f86d931ed37ee0e22eed1ea065d4bbfbbe17bdcb5afaf not found: ID does not exist"
Oct 31 14:02:51.827399 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.827311    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"]
Oct 31 14:02:51.833514 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.833409    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bgc5j"]
Oct 31 14:02:51.857345 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.853948    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-maas-api-tls\") pod \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\" (UID: \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\") "
Oct 31 14:02:51.857345 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.854016    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-kube-api-access-jgjqh\" (UniqueName: \"kubernetes.io/projected/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-kube-api-access-jgjqh\") pod \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\" (UID: \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\") "
Oct 31 14:02:51.857345 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.854156    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-metrics-certs\") pod \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\" (UID: \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\") "
Oct 31 14:02:51.857345 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.854249    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-openshift-service-ca\") pod \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\" (UID: \"3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e\") "
Oct 31 14:02:51.857345 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.856048    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-openshift-service-ca" pod "3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e" (UID: "3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:51.857345 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.856222    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:51.859809 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.859745    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-metrics-certs" pod "3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e" (UID: "3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:51.860071 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.860043    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-maas-api-tls" pod "3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e" (UID: "3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:51.860275 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.860248    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-kube-api-access-jgjqh" pod "3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e" (UID: "3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e"). InnerVolumeSpecName "kube-api-access-jgjqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:51.957258 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.957210    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:51.957258 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.957251    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:51.957522 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:51.957269    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgjqh\" (UniqueName: \"kubernetes.io/projected/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e-kube-api-access-jgjqh\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.119744 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.119698    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"]
Oct 31 14:02:52.126351 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.126315    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c82140-58c485798d-mtvcz"]
Oct 31 14:02:52.174529 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.174488    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" containerID="cri-o://17027bb59555c60ed44298aeee010cc90e48bf0bb551d4d3a4828f7a41ec0175" gracePeriod=45
Oct 31 14:02:52.319194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.319136    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": dial tcp 10.133.0.69:8000: connect: connection refused"
Oct 31 14:02:52.459333 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.459303    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 14:02:52.536847 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.531479    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e" path="/var/lib/kubelet/pods/3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e/volumes"
Oct 31 14:02:52.538507 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.538478    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a36003d-cda6-447e-bb9f-95f13e32c276" path="/var/lib/kubelet/pods/6a36003d-cda6-447e-bb9f-95f13e32c276/volumes"
Oct 31 14:02:52.563683 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.563646    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38522f51-cc0e-4026-a6f3-a9be0b429561-tls-certs\" (UniqueName: \"kubernetes.io/secret/38522f51-cc0e-4026-a6f3-a9be0b429561-tls-certs\") pod \"38522f51-cc0e-4026-a6f3-a9be0b429561\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") "
Oct 31 14:02:52.563878 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.563708    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-tmp-dir\") pod \"38522f51-cc0e-4026-a6f3-a9be0b429561\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") "
Oct 31 14:02:52.563878 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.563848    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-home\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-home\") pod \"38522f51-cc0e-4026-a6f3-a9be0b429561\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") "
Oct 31 14:02:52.564013 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.563990    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-tmp-dir" pod "38522f51-cc0e-4026-a6f3-a9be0b429561" (UID: "38522f51-cc0e-4026-a6f3-a9be0b429561"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:52.564180 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.564142    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-home" pod "38522f51-cc0e-4026-a6f3-a9be0b429561" (UID: "38522f51-cc0e-4026-a6f3-a9be0b429561"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:52.564348 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.564316    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-dshm\") pod \"38522f51-cc0e-4026-a6f3-a9be0b429561\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") "
Oct 31 14:02:52.564461 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.564444    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-model-cache\") pod \"38522f51-cc0e-4026-a6f3-a9be0b429561\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") "
Oct 31 14:02:52.564542 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.564527    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38522f51-cc0e-4026-a6f3-a9be0b429561-kube-api-access-bnw2n\" (UniqueName: \"kubernetes.io/projected/38522f51-cc0e-4026-a6f3-a9be0b429561-kube-api-access-bnw2n\") pod \"38522f51-cc0e-4026-a6f3-a9be0b429561\" (UID: \"38522f51-cc0e-4026-a6f3-a9be0b429561\") "
Oct 31 14:02:52.565028 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.565003    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.565118 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.565032    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.565191 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.565123    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-model-cache" pod "38522f51-cc0e-4026-a6f3-a9be0b429561" (UID: "38522f51-cc0e-4026-a6f3-a9be0b429561"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:52.566742 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.566705    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-dshm" pod "38522f51-cc0e-4026-a6f3-a9be0b429561" (UID: "38522f51-cc0e-4026-a6f3-a9be0b429561"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:52.567117 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.567089    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38522f51-cc0e-4026-a6f3-a9be0b429561-tls-certs" pod "38522f51-cc0e-4026-a6f3-a9be0b429561" (UID: "38522f51-cc0e-4026-a6f3-a9be0b429561"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:52.567254 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.567233    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38522f51-cc0e-4026-a6f3-a9be0b429561-kube-api-access-bnw2n" pod "38522f51-cc0e-4026-a6f3-a9be0b429561" (UID: "38522f51-cc0e-4026-a6f3-a9be0b429561"). InnerVolumeSpecName "kube-api-access-bnw2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:52.665856 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.665804    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.665856 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.665845    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38522f51-cc0e-4026-a6f3-a9be0b429561-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.665856 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.665858    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnw2n\" (UniqueName: \"kubernetes.io/projected/38522f51-cc0e-4026-a6f3-a9be0b429561-kube-api-access-bnw2n\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.665856 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.665868    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38522f51-cc0e-4026-a6f3-a9be0b429561-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.801509 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.801470    4660 generic.go:356] "Generic (PLEG): container finished" podID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerID="17027bb59555c60ed44298aeee010cc90e48bf0bb551d4d3a4828f7a41ec0175" exitCode=0
Oct 31 14:02:52.802091 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.801532    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"
Oct 31 14:02:52.802091 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.801571    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" event={"ID":"38522f51-cc0e-4026-a6f3-a9be0b429561","Type":"ContainerDied","Data":"17027bb59555c60ed44298aeee010cc90e48bf0bb551d4d3a4828f7a41ec0175"}
Oct 31 14:02:52.802091 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.801672    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj" event={"ID":"38522f51-cc0e-4026-a6f3-a9be0b429561","Type":"ContainerDied","Data":"dfa7f9549d9aa11cedb473c1eb3a64b98bf41750163660cf3436dae8b2d8529a"}
Oct 31 14:02:52.802091 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.801697    4660 scope.go:122] "RemoveContainer" containerID="17027bb59555c60ed44298aeee010cc90e48bf0bb551d4d3a4828f7a41ec0175"
Oct 31 14:02:52.814886 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.814862    4660 scope.go:122] "RemoveContainer" containerID="17027bb59555c60ed44298aeee010cc90e48bf0bb551d4d3a4828f7a41ec0175"
Oct 31 14:02:52.815221 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:02:52.815189    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17027bb59555c60ed44298aeee010cc90e48bf0bb551d4d3a4828f7a41ec0175\": container with ID starting with 17027bb59555c60ed44298aeee010cc90e48bf0bb551d4d3a4828f7a41ec0175 not found: ID does not exist" containerID="17027bb59555c60ed44298aeee010cc90e48bf0bb551d4d3a4828f7a41ec0175"
Oct 31 14:02:52.815364 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.815233    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17027bb59555c60ed44298aeee010cc90e48bf0bb551d4d3a4828f7a41ec0175"} err="failed to get container status \"17027bb59555c60ed44298aeee010cc90e48bf0bb551d4d3a4828f7a41ec0175\": rpc error: code = NotFound desc = could not find container \"17027bb59555c60ed44298aeee010cc90e48bf0bb551d4d3a4828f7a41ec0175\": container with ID starting with 17027bb59555c60ed44298aeee010cc90e48bf0bb551d4d3a4828f7a41ec0175 not found: ID does not exist"
Oct 31 14:02:52.830546 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.830493    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"]
Oct 31 14:02:52.837496 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.837463    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/distinct-w6-aabeef-kserve-84585dc6cf-xwcpj"]
Oct 31 14:02:52.845054 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:52.845018    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw" podUID="84174b11-1866-4f83-ad10-84a6288b53dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:54.526402 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:54.526367    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" path="/var/lib/kubelet/pods/38522f51-cc0e-4026-a6f3-a9be0b429561/volumes"
Oct 31 14:02:56.715316 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:56.715276    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"
Oct 31 14:02:56.810461 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:56.810424    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84174b11-1866-4f83-ad10-84a6288b53dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84174b11-1866-4f83-ad10-84a6288b53dd-plugins-config-volume\") pod \"84174b11-1866-4f83-ad10-84a6288b53dd\" (UID: \"84174b11-1866-4f83-ad10-84a6288b53dd\") "
Oct 31 14:02:56.810683 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:56.810648    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84174b11-1866-4f83-ad10-84a6288b53dd-kube-api-access-qk529\" (UniqueName: \"kubernetes.io/projected/84174b11-1866-4f83-ad10-84a6288b53dd-kube-api-access-qk529\") pod \"84174b11-1866-4f83-ad10-84a6288b53dd\" (UID: \"84174b11-1866-4f83-ad10-84a6288b53dd\") "
Oct 31 14:02:56.810846 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:56.810820    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84174b11-1866-4f83-ad10-84a6288b53dd-plugins-config-volume" pod "84174b11-1866-4f83-ad10-84a6288b53dd" (UID: "84174b11-1866-4f83-ad10-84a6288b53dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:56.810964 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:56.810946    4660 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84174b11-1866-4f83-ad10-84a6288b53dd-plugins-config-volume\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:56.812835 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:56.812809    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84174b11-1866-4f83-ad10-84a6288b53dd-kube-api-access-qk529" pod "84174b11-1866-4f83-ad10-84a6288b53dd" (UID: "84174b11-1866-4f83-ad10-84a6288b53dd"). InnerVolumeSpecName "kube-api-access-qk529". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:56.832726 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:56.832684    4660 generic.go:356] "Generic (PLEG): container finished" podID="84174b11-1866-4f83-ad10-84a6288b53dd" containerID="9d5ed341b35fee7a96a7b4bb55cab9e6e67ddc86fcd98e941140e6a8690ae92e" exitCode=0
Oct 31 14:02:56.832916 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:56.832735    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"
Oct 31 14:02:56.832916 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:56.832764    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw" event={"ID":"84174b11-1866-4f83-ad10-84a6288b53dd","Type":"ContainerDied","Data":"9d5ed341b35fee7a96a7b4bb55cab9e6e67ddc86fcd98e941140e6a8690ae92e"}
Oct 31 14:02:56.832916 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:56.832840    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw" event={"ID":"84174b11-1866-4f83-ad10-84a6288b53dd","Type":"ContainerDied","Data":"734c9990224c9a73d9bd7425284b2584b68b79a14696413791660bc396480876"}
Oct 31 14:02:56.832916 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:56.832871    4660 scope.go:122] "RemoveContainer" containerID="9d5ed341b35fee7a96a7b4bb55cab9e6e67ddc86fcd98e941140e6a8690ae92e"
Oct 31 14:02:56.843816 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:56.843797    4660 scope.go:122] "RemoveContainer" containerID="9d5ed341b35fee7a96a7b4bb55cab9e6e67ddc86fcd98e941140e6a8690ae92e"
Oct 31 14:02:56.844105 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:02:56.844085    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5ed341b35fee7a96a7b4bb55cab9e6e67ddc86fcd98e941140e6a8690ae92e\": container with ID starting with 9d5ed341b35fee7a96a7b4bb55cab9e6e67ddc86fcd98e941140e6a8690ae92e not found: ID does not exist" containerID="9d5ed341b35fee7a96a7b4bb55cab9e6e67ddc86fcd98e941140e6a8690ae92e"
Oct 31 14:02:56.844194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:56.844112    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5ed341b35fee7a96a7b4bb55cab9e6e67ddc86fcd98e941140e6a8690ae92e"} err="failed to get container status \"9d5ed341b35fee7a96a7b4bb55cab9e6e67ddc86fcd98e941140e6a8690ae92e\": rpc error: code = NotFound desc = could not find container \"9d5ed341b35fee7a96a7b4bb55cab9e6e67ddc86fcd98e941140e6a8690ae92e\": container with ID starting with 9d5ed341b35fee7a96a7b4bb55cab9e6e67ddc86fcd98e941140e6a8690ae92e not found: ID does not exist"
Oct 31 14:02:56.865349 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:56.865317    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"]
Oct 31 14:02:56.873154 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:56.873126    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c82140-575b9b7477-5qnmw"]
Oct 31 14:02:56.912099 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:56.912056    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk529\" (UniqueName: \"kubernetes.io/projected/84174b11-1866-4f83-ad10-84a6288b53dd-kube-api-access-qk529\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:58.527194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:02:58.527157    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84174b11-1866-4f83-ad10-84a6288b53dd" path="/var/lib/kubelet/pods/84174b11-1866-4f83-ad10-84a6288b53dd/volumes"
Oct 31 14:03:05.736842 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:05.736799    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"]
Oct 31 14:03:05.737403 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:05.737105    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w" podUID="8e133f4e-385e-46b4-ab50-db03976321e0" containerName="maas-api" containerID="cri-o://25675e9bc9c985c18c574b46ac180881ef881bdfdcea5c3f02885cfbc95c4d66" gracePeriod=30
Oct 31 14:03:05.918201 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:05.913639    4660 generic.go:356] "Generic (PLEG): container finished" podID="8e133f4e-385e-46b4-ab50-db03976321e0" containerID="25675e9bc9c985c18c574b46ac180881ef881bdfdcea5c3f02885cfbc95c4d66" exitCode=0
Oct 31 14:03:05.918201 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:05.913842    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w" event={"ID":"8e133f4e-385e-46b4-ab50-db03976321e0","Type":"ContainerDied","Data":"25675e9bc9c985c18c574b46ac180881ef881bdfdcea5c3f02885cfbc95c4d66"}
Oct 31 14:03:06.094779 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.094751    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 14:03:06.113797 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.113763    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e133f4e-385e-46b4-ab50-db03976321e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e133f4e-385e-46b4-ab50-db03976321e0-metrics-certs\") pod \"8e133f4e-385e-46b4-ab50-db03976321e0\" (UID: \"8e133f4e-385e-46b4-ab50-db03976321e0\") "
Oct 31 14:03:06.113797 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.113814    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e133f4e-385e-46b4-ab50-db03976321e0-kube-api-access-wmbqd\" (UniqueName: \"kubernetes.io/projected/8e133f4e-385e-46b4-ab50-db03976321e0-kube-api-access-wmbqd\") pod \"8e133f4e-385e-46b4-ab50-db03976321e0\" (UID: \"8e133f4e-385e-46b4-ab50-db03976321e0\") "
Oct 31 14:03:06.114653 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.113850    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e133f4e-385e-46b4-ab50-db03976321e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e133f4e-385e-46b4-ab50-db03976321e0-openshift-service-ca\") pod \"8e133f4e-385e-46b4-ab50-db03976321e0\" (UID: \"8e133f4e-385e-46b4-ab50-db03976321e0\") "
Oct 31 14:03:06.114653 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.113965    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e133f4e-385e-46b4-ab50-db03976321e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e133f4e-385e-46b4-ab50-db03976321e0-maas-api-tls\") pod \"8e133f4e-385e-46b4-ab50-db03976321e0\" (UID: \"8e133f4e-385e-46b4-ab50-db03976321e0\") "
Oct 31 14:03:06.114653 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.114221    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e133f4e-385e-46b4-ab50-db03976321e0-openshift-service-ca" pod "8e133f4e-385e-46b4-ab50-db03976321e0" (UID: "8e133f4e-385e-46b4-ab50-db03976321e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:03:06.114653 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.114510    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e133f4e-385e-46b4-ab50-db03976321e0-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:06.116254 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.116221    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e133f4e-385e-46b4-ab50-db03976321e0-kube-api-access-wmbqd" pod "8e133f4e-385e-46b4-ab50-db03976321e0" (UID: "8e133f4e-385e-46b4-ab50-db03976321e0"). InnerVolumeSpecName "kube-api-access-wmbqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:06.116254 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.116238    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e133f4e-385e-46b4-ab50-db03976321e0-metrics-certs" pod "8e133f4e-385e-46b4-ab50-db03976321e0" (UID: "8e133f4e-385e-46b4-ab50-db03976321e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:03:06.116493 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.116463    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e133f4e-385e-46b4-ab50-db03976321e0-maas-api-tls" pod "8e133f4e-385e-46b4-ab50-db03976321e0" (UID: "8e133f4e-385e-46b4-ab50-db03976321e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:03:06.216203 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.216163    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e133f4e-385e-46b4-ab50-db03976321e0-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:06.216203 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.216200    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmbqd\" (UniqueName: \"kubernetes.io/projected/8e133f4e-385e-46b4-ab50-db03976321e0-kube-api-access-wmbqd\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:06.216447 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.216215    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e133f4e-385e-46b4-ab50-db03976321e0-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:06.921086 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.921046    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"
Oct 31 14:03:06.921593 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.921084    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w" event={"ID":"8e133f4e-385e-46b4-ab50-db03976321e0","Type":"ContainerDied","Data":"66d5a7d0c44c144af7398f9a47565ff5bea096754b09d9d8388d1db66651d7a2"}
Oct 31 14:03:06.921593 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.921157    4660 scope.go:122] "RemoveContainer" containerID="25675e9bc9c985c18c574b46ac180881ef881bdfdcea5c3f02885cfbc95c4d66"
Oct 31 14:03:06.944069 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.944035    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"]
Oct 31 14:03:06.952779 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:06.952747    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-aabeef-5d8db86c67-dsm5w"]
Oct 31 14:03:08.530712 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:08.530662    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e133f4e-385e-46b4-ab50-db03976321e0" path="/var/lib/kubelet/pods/8e133f4e-385e-46b4-ab50-db03976321e0/volumes"
Oct 31 14:03:15.221344 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:15.219051    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"]
Oct 31 14:03:15.221344 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:15.219324    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq" podUID="f696d368-e075-482d-8e6f-da424076afa0" containerName="istio-proxy" containerID="cri-o://2b07b498d9efd10c0642cb2af5a8def0d0aad2bb6ea163e27931efd44f4fdb66" gracePeriod=30
Oct 31 14:03:20.487992 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.487959    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 14:03:20.595567 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.595468    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-workload-certs\") pod \"f696d368-e075-482d-8e6f-da424076afa0\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") "
Oct 31 14:03:20.595567 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.595550    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f696d368-e075-482d-8e6f-da424076afa0-kube-api-access-5smzr\" (UniqueName: \"kubernetes.io/projected/f696d368-e075-482d-8e6f-da424076afa0-kube-api-access-5smzr\") pod \"f696d368-e075-482d-8e6f-da424076afa0\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") "
Oct 31 14:03:20.595823 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.595579    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-istio-data\") pod \"f696d368-e075-482d-8e6f-da424076afa0\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") "
Oct 31 14:03:20.595823 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.595632    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f696d368-e075-482d-8e6f-da424076afa0-istio-token\" (UniqueName: \"kubernetes.io/projected/f696d368-e075-482d-8e6f-da424076afa0-istio-token\") pod \"f696d368-e075-482d-8e6f-da424076afa0\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") "
Oct 31 14:03:20.595823 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.595723    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-workload-socket\") pod \"f696d368-e075-482d-8e6f-da424076afa0\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") "
Oct 31 14:03:20.595984 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.595838    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f696d368-e075-482d-8e6f-da424076afa0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f696d368-e075-482d-8e6f-da424076afa0-istio-podinfo\") pod \"f696d368-e075-482d-8e6f-da424076afa0\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") "
Oct 31 14:03:20.595984 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.595870    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-workload-certs" pod "f696d368-e075-482d-8e6f-da424076afa0" (UID: "f696d368-e075-482d-8e6f-da424076afa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:20.595984 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.595915    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-istio-envoy\") pod \"f696d368-e075-482d-8e6f-da424076afa0\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") "
Oct 31 14:03:20.595984 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.595955    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f696d368-e075-482d-8e6f-da424076afa0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f696d368-e075-482d-8e6f-da424076afa0-istiod-ca-cert\") pod \"f696d368-e075-482d-8e6f-da424076afa0\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") "
Oct 31 14:03:20.596226 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.596011    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-credential-socket\") pod \"f696d368-e075-482d-8e6f-da424076afa0\" (UID: \"f696d368-e075-482d-8e6f-da424076afa0\") "
Oct 31 14:03:20.596226 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.596055    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-istio-data" pod "f696d368-e075-482d-8e6f-da424076afa0" (UID: "f696d368-e075-482d-8e6f-da424076afa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:20.596226 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.596099    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-workload-socket" pod "f696d368-e075-482d-8e6f-da424076afa0" (UID: "f696d368-e075-482d-8e6f-da424076afa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:20.596608 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.596535    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-credential-socket" pod "f696d368-e075-482d-8e6f-da424076afa0" (UID: "f696d368-e075-482d-8e6f-da424076afa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:20.596608 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.596570    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.596608 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.596595    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.596792 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.596610    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.596792 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.596569    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f696d368-e075-482d-8e6f-da424076afa0-istiod-ca-cert" pod "f696d368-e075-482d-8e6f-da424076afa0" (UID: "f696d368-e075-482d-8e6f-da424076afa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:03:20.598089 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.598063    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f696d368-e075-482d-8e6f-da424076afa0-kube-api-access-5smzr" pod "f696d368-e075-482d-8e6f-da424076afa0" (UID: "f696d368-e075-482d-8e6f-da424076afa0"). InnerVolumeSpecName "kube-api-access-5smzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:20.598261 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.598230    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-istio-envoy" pod "f696d368-e075-482d-8e6f-da424076afa0" (UID: "f696d368-e075-482d-8e6f-da424076afa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:20.598427 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.598345    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f696d368-e075-482d-8e6f-da424076afa0-istio-token" pod "f696d368-e075-482d-8e6f-da424076afa0" (UID: "f696d368-e075-482d-8e6f-da424076afa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:20.598497 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.598464    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f696d368-e075-482d-8e6f-da424076afa0-istio-podinfo" pod "f696d368-e075-482d-8e6f-da424076afa0" (UID: "f696d368-e075-482d-8e6f-da424076afa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:03:20.697729 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.697687    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.697729 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.697724    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5smzr\" (UniqueName: \"kubernetes.io/projected/f696d368-e075-482d-8e6f-da424076afa0-kube-api-access-5smzr\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.697729 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.697734    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f696d368-e075-482d-8e6f-da424076afa0-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.698000 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.697750    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f696d368-e075-482d-8e6f-da424076afa0-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.698000 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.697761    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f696d368-e075-482d-8e6f-da424076afa0-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.698000 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:20.697773    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f696d368-e075-482d-8e6f-da424076afa0-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.739855 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:03:20.739811    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:03:21.016937 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:21.016901    4660 generic.go:356] "Generic (PLEG): container finished" podID="f696d368-e075-482d-8e6f-da424076afa0" containerID="2b07b498d9efd10c0642cb2af5a8def0d0aad2bb6ea163e27931efd44f4fdb66" exitCode=0
Oct 31 14:03:21.017230 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:21.016964    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"
Oct 31 14:03:21.017230 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:21.016972    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq" event={"ID":"f696d368-e075-482d-8e6f-da424076afa0","Type":"ContainerDied","Data":"2b07b498d9efd10c0642cb2af5a8def0d0aad2bb6ea163e27931efd44f4fdb66"}
Oct 31 14:03:21.017230 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:21.017087    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq" event={"ID":"f696d368-e075-482d-8e6f-da424076afa0","Type":"ContainerDied","Data":"7abcf567ec8d4667468d150706d2ca72cdcc0e191a848b97d159d7c831575b5a"}
Oct 31 14:03:21.017230 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:21.017118    4660 scope.go:122] "RemoveContainer" containerID="2b07b498d9efd10c0642cb2af5a8def0d0aad2bb6ea163e27931efd44f4fdb66"
Oct 31 14:03:21.034587 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:21.034557    4660 scope.go:122] "RemoveContainer" containerID="2b07b498d9efd10c0642cb2af5a8def0d0aad2bb6ea163e27931efd44f4fdb66"
Oct 31 14:03:21.036185 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:03:21.036143    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b07b498d9efd10c0642cb2af5a8def0d0aad2bb6ea163e27931efd44f4fdb66\": container with ID starting with 2b07b498d9efd10c0642cb2af5a8def0d0aad2bb6ea163e27931efd44f4fdb66 not found: ID does not exist" containerID="2b07b498d9efd10c0642cb2af5a8def0d0aad2bb6ea163e27931efd44f4fdb66"
Oct 31 14:03:21.036392 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:21.036195    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b07b498d9efd10c0642cb2af5a8def0d0aad2bb6ea163e27931efd44f4fdb66"} err="failed to get container status \"2b07b498d9efd10c0642cb2af5a8def0d0aad2bb6ea163e27931efd44f4fdb66\": rpc error: code = NotFound desc = could not find container \"2b07b498d9efd10c0642cb2af5a8def0d0aad2bb6ea163e27931efd44f4fdb66\": container with ID starting with 2b07b498d9efd10c0642cb2af5a8def0d0aad2bb6ea163e27931efd44f4fdb66 not found: ID does not exist"
Oct 31 14:03:21.050881 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:21.050845    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"]
Oct 31 14:03:21.057344 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:21.057314    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-f8ccq"]
Oct 31 14:03:21.382688 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:21.382657    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 14:03:21.386616 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:21.386583    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 14:03:21.389631 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:21.389344    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 14:03:21.393688 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:21.393667    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 14:03:21.398701 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:21.398676    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 14:03:21.402957 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:21.402933    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 14:03:22.527538 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:22.527495    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f696d368-e075-482d-8e6f-da424076afa0" path="/var/lib/kubelet/pods/f696d368-e075-482d-8e6f-da424076afa0/volumes"
Oct 31 14:03:50.734818 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:03:50.734781    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:03:54.286636 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:54.286587    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"]
Oct 31 14:03:54.289198 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:54.287476    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9" podUID="44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" containerName="istio-proxy" containerID="cri-o://7c50b79f3f6caacc98982fe5bb7ab1c4adeea14c806aca1b0589f52ec2c4a0a7" gracePeriod=30
Oct 31 14:03:59.568944 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.568908    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:03:59.716968 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.716934    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-token\" (UniqueName: \"kubernetes.io/projected/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-token\") pod \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") "
Oct 31 14:03:59.717261 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.716989    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-credential-socket\") pod \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") "
Oct 31 14:03:59.717261 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.717034    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-data\") pod \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") "
Oct 31 14:03:59.717261 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.717124    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-podinfo\") pod \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") "
Oct 31 14:03:59.717261 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.717171    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-kube-api-access-hwh6r\" (UniqueName: \"kubernetes.io/projected/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-kube-api-access-hwh6r\") pod \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") "
Oct 31 14:03:59.717261 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.717204    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istiod-ca-cert\") pod \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") "
Oct 31 14:03:59.717726 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.717329    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-envoy\") pod \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") "
Oct 31 14:03:59.717726 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.717362    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-data" pod "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" (UID: "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:59.717726 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.717375    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-workload-socket\") pod \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") "
Oct 31 14:03:59.717726 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.717382    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-credential-socket" pod "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" (UID: "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:59.717726 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.717678    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-workload-socket" pod "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" (UID: "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:59.717998 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.717757    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-workload-certs\") pod \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\" (UID: \"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c\") "
Oct 31 14:03:59.717998 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.717781    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-workload-certs" pod "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" (UID: "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:59.717998 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.717804    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istiod-ca-cert" pod "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" (UID: "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:03:59.718217 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.718196    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:59.718278 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.718225    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:59.718278 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.718242    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:59.718278 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.718258    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:59.718278 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.718273    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:59.719785 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.719755    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-envoy" pod "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" (UID: "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:59.720053 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.720021    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-kube-api-access-hwh6r" pod "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" (UID: "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c"). InnerVolumeSpecName "kube-api-access-hwh6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:59.720169 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.720062    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-token" pod "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" (UID: "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:59.720221 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.720193    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-podinfo" pod "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" (UID: "44fe7652-1fd9-470e-b8f9-a3cbcc061a8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:03:59.819244 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.819156    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:59.819244 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.819193    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwh6r\" (UniqueName: \"kubernetes.io/projected/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-kube-api-access-hwh6r\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:59.819244 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.819204    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:59.819244 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:03:59.819213    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:00.287770 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:00.287733    4660 generic.go:356] "Generic (PLEG): container finished" podID="44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" containerID="7c50b79f3f6caacc98982fe5bb7ab1c4adeea14c806aca1b0589f52ec2c4a0a7" exitCode=0
Oct 31 14:04:00.287990 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:00.287787    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"
Oct 31 14:04:00.287990 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:00.287808    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9" event={"ID":"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c","Type":"ContainerDied","Data":"7c50b79f3f6caacc98982fe5bb7ab1c4adeea14c806aca1b0589f52ec2c4a0a7"}
Oct 31 14:04:00.287990 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:00.287876    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9" event={"ID":"44fe7652-1fd9-470e-b8f9-a3cbcc061a8c","Type":"ContainerDied","Data":"f5c4703fece358416b9c967b81030bd7451169e5d0ed7a0e72dc66671f1873a9"}
Oct 31 14:04:00.287990 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:00.287905    4660 scope.go:122] "RemoveContainer" containerID="7c50b79f3f6caacc98982fe5bb7ab1c4adeea14c806aca1b0589f52ec2c4a0a7"
Oct 31 14:04:00.298951 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:00.298917    4660 scope.go:122] "RemoveContainer" containerID="7c50b79f3f6caacc98982fe5bb7ab1c4adeea14c806aca1b0589f52ec2c4a0a7"
Oct 31 14:04:00.299244 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:04:00.299223    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c50b79f3f6caacc98982fe5bb7ab1c4adeea14c806aca1b0589f52ec2c4a0a7\": container with ID starting with 7c50b79f3f6caacc98982fe5bb7ab1c4adeea14c806aca1b0589f52ec2c4a0a7 not found: ID does not exist" containerID="7c50b79f3f6caacc98982fe5bb7ab1c4adeea14c806aca1b0589f52ec2c4a0a7"
Oct 31 14:04:00.299340 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:00.299254    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c50b79f3f6caacc98982fe5bb7ab1c4adeea14c806aca1b0589f52ec2c4a0a7"} err="failed to get container status \"7c50b79f3f6caacc98982fe5bb7ab1c4adeea14c806aca1b0589f52ec2c4a0a7\": rpc error: code = NotFound desc = could not find container \"7c50b79f3f6caacc98982fe5bb7ab1c4adeea14c806aca1b0589f52ec2c4a0a7\": container with ID starting with 7c50b79f3f6caacc98982fe5bb7ab1c4adeea14c806aca1b0589f52ec2c4a0a7 not found: ID does not exist"
Oct 31 14:04:00.317814 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:00.317775    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"]
Oct 31 14:04:00.329546 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:00.329512    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-vh9q9"]
Oct 31 14:04:00.534618 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:00.534577    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" path="/var/lib/kubelet/pods/44fe7652-1fd9-470e-b8f9-a3cbcc061a8c/volumes"
Oct 31 14:04:20.733407 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:04:20.733363    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:04:20.941329 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:20.941276    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"]
Oct 31 14:04:20.951614 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:20.951582    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"]
Oct 31 14:04:20.959332 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:20.959280    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"]
Oct 31 14:04:35.963031 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:35.962980    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" containerID="cri-o://b84ba38848d2d9fd16e316b0fc0c62eae71410c020211c01e624c8e7a5d166e7" gracePeriod=45
Oct 31 14:04:35.972373 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:35.972284    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" containerID="cri-o://e4a945cfaa7729b72fba229fb6cd54809980904cff7b87ad50237f5d6ba5a09c" gracePeriod=45
Oct 31 14:04:35.980678 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:35.980644    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main" containerID="cri-o://526fd02de6133614fba2e1aeb172daec9d81149c56e2cff8bacb5ea026e624e1" gracePeriod=45
Oct 31 14:04:36.141803 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.141759    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": dial tcp 10.133.0.71:8000: connect: connection refused"
Oct 31 14:04:36.263236 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.262842    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 14:04:36.266780 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.266753    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 14:04:36.294001 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.293966    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 14:04:36.399777 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.399738    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd80067-027a-40ab-be2f-63e7597a72f3-kube-api-access-g96jv\" (UniqueName: \"kubernetes.io/projected/0fd80067-027a-40ab-be2f-63e7597a72f3-kube-api-access-g96jv\") pod \"0fd80067-027a-40ab-be2f-63e7597a72f3\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") "
Oct 31 14:04:36.399966 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.399783    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-model-cache\") pod \"0fd80067-027a-40ab-be2f-63e7597a72f3\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") "
Oct 31 14:04:36.399966 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.399838    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-model-cache\") pod \"f68b0a7c-08d8-42e2-8b56-f81b58386500\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") "
Oct 31 14:04:36.399966 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.399950    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd80067-027a-40ab-be2f-63e7597a72f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd80067-027a-40ab-be2f-63e7597a72f3-tls-certs\") pod \"0fd80067-027a-40ab-be2f-63e7597a72f3\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") "
Oct 31 14:04:36.400173 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400013    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-dshm\") pod \"0fd80067-027a-40ab-be2f-63e7597a72f3\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") "
Oct 31 14:04:36.400173 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400050    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-tmp-dir\") pod \"f68b0a7c-08d8-42e2-8b56-f81b58386500\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") "
Oct 31 14:04:36.400173 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400094    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-dshm\") pod \"f68b0a7c-08d8-42e2-8b56-f81b58386500\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") "
Oct 31 14:04:36.400173 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400107    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-model-cache" pod "f68b0a7c-08d8-42e2-8b56-f81b58386500" (UID: "f68b0a7c-08d8-42e2-8b56-f81b58386500"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:36.400173 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400125    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f68b0a7c-08d8-42e2-8b56-f81b58386500-tls-certs\" (UniqueName: \"kubernetes.io/secret/f68b0a7c-08d8-42e2-8b56-f81b58386500-tls-certs\") pod \"f68b0a7c-08d8-42e2-8b56-f81b58386500\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") "
Oct 31 14:04:36.400480 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400192    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-home\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-home\") pod \"f68b0a7c-08d8-42e2-8b56-f81b58386500\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") "
Oct 31 14:04:36.400480 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400194    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-model-cache" pod "0fd80067-027a-40ab-be2f-63e7597a72f3" (UID: "0fd80067-027a-40ab-be2f-63e7597a72f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:36.400480 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400240    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-home\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-home\") pod \"0fd80067-027a-40ab-be2f-63e7597a72f3\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") "
Oct 31 14:04:36.400480 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400304    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-tmp-dir\") pod \"0fd80067-027a-40ab-be2f-63e7597a72f3\" (UID: \"0fd80067-027a-40ab-be2f-63e7597a72f3\") "
Oct 31 14:04:36.400480 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400344    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-tmp-dir\") pod \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") "
Oct 31 14:04:36.400480 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400407    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-home\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-home\") pod \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") "
Oct 31 14:04:36.400480 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400453    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f68b0a7c-08d8-42e2-8b56-f81b58386500-kube-api-access-bpmsh\" (UniqueName: \"kubernetes.io/projected/f68b0a7c-08d8-42e2-8b56-f81b58386500-kube-api-access-bpmsh\") pod \"f68b0a7c-08d8-42e2-8b56-f81b58386500\" (UID: \"f68b0a7c-08d8-42e2-8b56-f81b58386500\") "
Oct 31 14:04:36.400480 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400482    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-dshm\") pod \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") "
Oct 31 14:04:36.400962 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400485    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-tmp-dir" pod "f68b0a7c-08d8-42e2-8b56-f81b58386500" (UID: "f68b0a7c-08d8-42e2-8b56-f81b58386500"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:36.400962 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400520    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bce80f7f-51e8-46f3-add6-37bbb3a0adba-tls-certs\" (UniqueName: \"kubernetes.io/secret/bce80f7f-51e8-46f3-add6-37bbb3a0adba-tls-certs\") pod \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") "
Oct 31 14:04:36.400962 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400587    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce80f7f-51e8-46f3-add6-37bbb3a0adba-kube-api-access-zt4qm\" (UniqueName: \"kubernetes.io/projected/bce80f7f-51e8-46f3-add6-37bbb3a0adba-kube-api-access-zt4qm\") pod \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") "
Oct 31 14:04:36.400962 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.400657    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-model-cache\") pod \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\" (UID: \"bce80f7f-51e8-46f3-add6-37bbb3a0adba\") "
Oct 31 14:04:36.404051 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.404016    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-home" pod "bce80f7f-51e8-46f3-add6-37bbb3a0adba" (UID: "bce80f7f-51e8-46f3-add6-37bbb3a0adba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:36.404191 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.404068    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-home" pod "0fd80067-027a-40ab-be2f-63e7597a72f3" (UID: "0fd80067-027a-40ab-be2f-63e7597a72f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:36.404191 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.404138    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd80067-027a-40ab-be2f-63e7597a72f3-tls-certs" pod "0fd80067-027a-40ab-be2f-63e7597a72f3" (UID: "0fd80067-027a-40ab-be2f-63e7597a72f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:04:36.404328 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.404202    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-dshm" pod "f68b0a7c-08d8-42e2-8b56-f81b58386500" (UID: "f68b0a7c-08d8-42e2-8b56-f81b58386500"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:36.404328 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.404240    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f68b0a7c-08d8-42e2-8b56-f81b58386500-tls-certs" pod "f68b0a7c-08d8-42e2-8b56-f81b58386500" (UID: "f68b0a7c-08d8-42e2-8b56-f81b58386500"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:04:36.404540 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.404521    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-tmp-dir" pod "0fd80067-027a-40ab-be2f-63e7597a72f3" (UID: "0fd80067-027a-40ab-be2f-63e7597a72f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:36.404979 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.404951    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-tmp-dir" pod "bce80f7f-51e8-46f3-add6-37bbb3a0adba" (UID: "bce80f7f-51e8-46f3-add6-37bbb3a0adba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:36.405152 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.405109    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-dshm" pod "0fd80067-027a-40ab-be2f-63e7597a72f3" (UID: "0fd80067-027a-40ab-be2f-63e7597a72f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:36.407328 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.405380    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd80067-027a-40ab-be2f-63e7597a72f3-kube-api-access-g96jv" pod "0fd80067-027a-40ab-be2f-63e7597a72f3" (UID: "0fd80067-027a-40ab-be2f-63e7597a72f3"). InnerVolumeSpecName "kube-api-access-g96jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:04:36.407328 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.405529    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-home" pod "f68b0a7c-08d8-42e2-8b56-f81b58386500" (UID: "f68b0a7c-08d8-42e2-8b56-f81b58386500"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:36.407328 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.406497    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.407328 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.406547    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.407328 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.406564    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.407328 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.407287    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce80f7f-51e8-46f3-add6-37bbb3a0adba-kube-api-access-zt4qm" pod "bce80f7f-51e8-46f3-add6-37bbb3a0adba" (UID: "bce80f7f-51e8-46f3-add6-37bbb3a0adba"). InnerVolumeSpecName "kube-api-access-zt4qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:04:36.415856 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.415818    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-dshm" pod "bce80f7f-51e8-46f3-add6-37bbb3a0adba" (UID: "bce80f7f-51e8-46f3-add6-37bbb3a0adba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:36.417266 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.417238    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-model-cache" pod "bce80f7f-51e8-46f3-add6-37bbb3a0adba" (UID: "bce80f7f-51e8-46f3-add6-37bbb3a0adba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:36.417758 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.417715    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce80f7f-51e8-46f3-add6-37bbb3a0adba-tls-certs" pod "bce80f7f-51e8-46f3-add6-37bbb3a0adba" (UID: "bce80f7f-51e8-46f3-add6-37bbb3a0adba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:04:36.422760 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.422414    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68b0a7c-08d8-42e2-8b56-f81b58386500-kube-api-access-bpmsh" pod "f68b0a7c-08d8-42e2-8b56-f81b58386500" (UID: "f68b0a7c-08d8-42e2-8b56-f81b58386500"). InnerVolumeSpecName "kube-api-access-bpmsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:04:36.507653 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.507562    4660 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-model-cache\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.507653 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.507596    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g96jv\" (UniqueName: \"kubernetes.io/projected/0fd80067-027a-40ab-be2f-63e7597a72f3-kube-api-access-g96jv\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.507653 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.507607    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd80067-027a-40ab-be2f-63e7597a72f3-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.507653 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.507616    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.507653 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.507624    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.507653 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.507632    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f68b0a7c-08d8-42e2-8b56-f81b58386500-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.507653 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.507639    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f68b0a7c-08d8-42e2-8b56-f81b58386500-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.507653 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.507646    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.507653 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.507654    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd80067-027a-40ab-be2f-63e7597a72f3-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.507653 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.507661    4660 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-tmp-dir\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.508098 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.507669    4660 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-home\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.508098 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.507677    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpmsh\" (UniqueName: \"kubernetes.io/projected/f68b0a7c-08d8-42e2-8b56-f81b58386500-kube-api-access-bpmsh\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.508098 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.507685    4660 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce80f7f-51e8-46f3-add6-37bbb3a0adba-dshm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.508098 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.507694    4660 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce80f7f-51e8-46f3-add6-37bbb3a0adba-tls-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.508098 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.507702    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt4qm\" (UniqueName: \"kubernetes.io/projected/bce80f7f-51e8-46f3-add6-37bbb3a0adba-kube-api-access-zt4qm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:36.515553 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.515520    4660 generic.go:356] "Generic (PLEG): container finished" podID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerID="b84ba38848d2d9fd16e316b0fc0c62eae71410c020211c01e624c8e7a5d166e7" exitCode=0
Oct 31 14:04:36.515711 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.515586    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"
Oct 31 14:04:36.515711 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.515607    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" event={"ID":"f68b0a7c-08d8-42e2-8b56-f81b58386500","Type":"ContainerDied","Data":"b84ba38848d2d9fd16e316b0fc0c62eae71410c020211c01e624c8e7a5d166e7"}
Oct 31 14:04:36.515711 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.515654    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl" event={"ID":"f68b0a7c-08d8-42e2-8b56-f81b58386500","Type":"ContainerDied","Data":"c4ec7ce38f57a235eaacbcd28361d22916db7ae9dce213e0d743255604f73158"}
Oct 31 14:04:36.515711 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.515671    4660 scope.go:122] "RemoveContainer" containerID="b84ba38848d2d9fd16e316b0fc0c62eae71410c020211c01e624c8e7a5d166e7"
Oct 31 14:04:36.517230 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.517203    4660 generic.go:356] "Generic (PLEG): container finished" podID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerID="e4a945cfaa7729b72fba229fb6cd54809980904cff7b87ad50237f5d6ba5a09c" exitCode=0
Oct 31 14:04:36.517436 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.517274    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"
Oct 31 14:04:36.517436 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.517314    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" event={"ID":"bce80f7f-51e8-46f3-add6-37bbb3a0adba","Type":"ContainerDied","Data":"e4a945cfaa7729b72fba229fb6cd54809980904cff7b87ad50237f5d6ba5a09c"}
Oct 31 14:04:36.517436 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.517364    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" event={"ID":"bce80f7f-51e8-46f3-add6-37bbb3a0adba","Type":"ContainerDied","Data":"a18aec24480098a5ca4afac7f547eb16ae9a82bdd452e11aa74ab5192e34b25a"}
Oct 31 14:04:36.518887 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.518863    4660 generic.go:356] "Generic (PLEG): container finished" podID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerID="526fd02de6133614fba2e1aeb172daec9d81149c56e2cff8bacb5ea026e624e1" exitCode=0
Oct 31 14:04:36.519008 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.518992    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" event={"ID":"0fd80067-027a-40ab-be2f-63e7597a72f3","Type":"ContainerDied","Data":"526fd02de6133614fba2e1aeb172daec9d81149c56e2cff8bacb5ea026e624e1"}
Oct 31 14:04:36.519070 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.519023    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2" event={"ID":"0fd80067-027a-40ab-be2f-63e7597a72f3","Type":"ContainerDied","Data":"278fdb8993d9d39c6d8393526550a9f931f7bf6139a919433d4493264decb83c"}
Oct 31 14:04:36.519118 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.519095    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"
Oct 31 14:04:36.528267 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.528245    4660 scope.go:122] "RemoveContainer" containerID="b84ba38848d2d9fd16e316b0fc0c62eae71410c020211c01e624c8e7a5d166e7"
Oct 31 14:04:36.528679 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:04:36.528615    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b84ba38848d2d9fd16e316b0fc0c62eae71410c020211c01e624c8e7a5d166e7\": container with ID starting with b84ba38848d2d9fd16e316b0fc0c62eae71410c020211c01e624c8e7a5d166e7 not found: ID does not exist" containerID="b84ba38848d2d9fd16e316b0fc0c62eae71410c020211c01e624c8e7a5d166e7"
Oct 31 14:04:36.528679 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.528653    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84ba38848d2d9fd16e316b0fc0c62eae71410c020211c01e624c8e7a5d166e7"} err="failed to get container status \"b84ba38848d2d9fd16e316b0fc0c62eae71410c020211c01e624c8e7a5d166e7\": rpc error: code = NotFound desc = could not find container \"b84ba38848d2d9fd16e316b0fc0c62eae71410c020211c01e624c8e7a5d166e7\": container with ID starting with b84ba38848d2d9fd16e316b0fc0c62eae71410c020211c01e624c8e7a5d166e7 not found: ID does not exist"
Oct 31 14:04:36.528679 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.528677    4660 scope.go:122] "RemoveContainer" containerID="e4a945cfaa7729b72fba229fb6cd54809980904cff7b87ad50237f5d6ba5a09c"
Oct 31 14:04:36.541562 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.541540    4660 scope.go:122] "RemoveContainer" containerID="e4a945cfaa7729b72fba229fb6cd54809980904cff7b87ad50237f5d6ba5a09c"
Oct 31 14:04:36.541848 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:04:36.541825    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a945cfaa7729b72fba229fb6cd54809980904cff7b87ad50237f5d6ba5a09c\": container with ID starting with e4a945cfaa7729b72fba229fb6cd54809980904cff7b87ad50237f5d6ba5a09c not found: ID does not exist" containerID="e4a945cfaa7729b72fba229fb6cd54809980904cff7b87ad50237f5d6ba5a09c"
Oct 31 14:04:36.541898 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.541857    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a945cfaa7729b72fba229fb6cd54809980904cff7b87ad50237f5d6ba5a09c"} err="failed to get container status \"e4a945cfaa7729b72fba229fb6cd54809980904cff7b87ad50237f5d6ba5a09c\": rpc error: code = NotFound desc = could not find container \"e4a945cfaa7729b72fba229fb6cd54809980904cff7b87ad50237f5d6ba5a09c\": container with ID starting with e4a945cfaa7729b72fba229fb6cd54809980904cff7b87ad50237f5d6ba5a09c not found: ID does not exist"
Oct 31 14:04:36.541898 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.541874    4660 scope.go:122] "RemoveContainer" containerID="526fd02de6133614fba2e1aeb172daec9d81149c56e2cff8bacb5ea026e624e1"
Oct 31 14:04:36.551021 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.550994    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"]
Oct 31 14:04:36.553373 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.553350    4660 scope.go:122] "RemoveContainer" containerID="526fd02de6133614fba2e1aeb172daec9d81149c56e2cff8bacb5ea026e624e1"
Oct 31 14:04:36.553714 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:04:36.553692    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"526fd02de6133614fba2e1aeb172daec9d81149c56e2cff8bacb5ea026e624e1\": container with ID starting with 526fd02de6133614fba2e1aeb172daec9d81149c56e2cff8bacb5ea026e624e1 not found: ID does not exist" containerID="526fd02de6133614fba2e1aeb172daec9d81149c56e2cff8bacb5ea026e624e1"
Oct 31 14:04:36.553780 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.553724    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526fd02de6133614fba2e1aeb172daec9d81149c56e2cff8bacb5ea026e624e1"} err="failed to get container status \"526fd02de6133614fba2e1aeb172daec9d81149c56e2cff8bacb5ea026e624e1\": rpc error: code = NotFound desc = could not find container \"526fd02de6133614fba2e1aeb172daec9d81149c56e2cff8bacb5ea026e624e1\": container with ID starting with 526fd02de6133614fba2e1aeb172daec9d81149c56e2cff8bacb5ea026e624e1 not found: ID does not exist"
Oct 31 14:04:36.555086 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.555065    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx"]
Oct 31 14:04:36.571235 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.571206    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"]
Oct 31 14:04:36.576361 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.576335    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/distinct-w4-8b2f0a-kserve-6b7f4d75fb-6m2bl"]
Oct 31 14:04:36.589791 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.589757    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"]
Oct 31 14:04:36.594102 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:36.594076    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/simulator-w4-8b2f0a-kserve-5f7975bfd4-24gq2"]
Oct 31 14:04:37.176048 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:37.175999    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/premium-w4-8b2f0a-kserve-5b84ff6949-2pbnx" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 14:04:38.527438 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:38.527398    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" path="/var/lib/kubelet/pods/0fd80067-027a-40ab-be2f-63e7597a72f3/volumes"
Oct 31 14:04:38.527982 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:38.527960    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" path="/var/lib/kubelet/pods/bce80f7f-51e8-46f3-add6-37bbb3a0adba/volumes"
Oct 31 14:04:38.528550 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:38.528531    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" path="/var/lib/kubelet/pods/f68b0a7c-08d8-42e2-8b56-f81b58386500/volumes"
Oct 31 14:04:49.849266 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:49.849229    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"]
Oct 31 14:04:49.849859 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:49.849502    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerName="istio-proxy" containerID="cri-o://cdf87235268786cf2bc510827622283b3d5f06ad016a2cd81c385b3bc5be62f5" gracePeriod=30
Oct 31 14:04:50.734818 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:04:50.734770    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:04:52.467932 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:52.467886    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56949457b6-mx968"]
Oct 31 14:04:52.468483 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:52.468150    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-56949457b6-mx968" podUID="5cf14527-ef89-4011-bad8-4ccd4bc245ec" containerName="manager" containerID="cri-o://200bf4f0eab1c415e741b153bb49adf1cbd838c9a8c6d25f9af93eb96fd23f61" gracePeriod=30
Oct 31 14:04:52.635116 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:52.635073    4660 generic.go:356] "Generic (PLEG): container finished" podID="5cf14527-ef89-4011-bad8-4ccd4bc245ec" containerID="200bf4f0eab1c415e741b153bb49adf1cbd838c9a8c6d25f9af93eb96fd23f61" exitCode=0
Oct 31 14:04:52.635410 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:52.635140    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56949457b6-mx968" event={"ID":"5cf14527-ef89-4011-bad8-4ccd4bc245ec","Type":"ContainerDied","Data":"200bf4f0eab1c415e741b153bb49adf1cbd838c9a8c6d25f9af93eb96fd23f61"}
Oct 31 14:04:52.733157 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:52.733096    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56949457b6-mx968"
Oct 31 14:04:52.864521 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:52.864481    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cf14527-ef89-4011-bad8-4ccd4bc245ec-webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cf14527-ef89-4011-bad8-4ccd4bc245ec-webhook-cert\") pod \"5cf14527-ef89-4011-bad8-4ccd4bc245ec\" (UID: \"5cf14527-ef89-4011-bad8-4ccd4bc245ec\") "
Oct 31 14:04:52.864738 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:52.864539    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf14527-ef89-4011-bad8-4ccd4bc245ec-kube-api-access-9wvtm\" (UniqueName: \"kubernetes.io/projected/5cf14527-ef89-4011-bad8-4ccd4bc245ec-kube-api-access-9wvtm\") pod \"5cf14527-ef89-4011-bad8-4ccd4bc245ec\" (UID: \"5cf14527-ef89-4011-bad8-4ccd4bc245ec\") "
Oct 31 14:04:52.864738 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:52.864587    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cf14527-ef89-4011-bad8-4ccd4bc245ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cf14527-ef89-4011-bad8-4ccd4bc245ec-metrics-certs\") pod \"5cf14527-ef89-4011-bad8-4ccd4bc245ec\" (UID: \"5cf14527-ef89-4011-bad8-4ccd4bc245ec\") "
Oct 31 14:04:52.867218 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:52.867183    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cf14527-ef89-4011-bad8-4ccd4bc245ec-metrics-certs" pod "5cf14527-ef89-4011-bad8-4ccd4bc245ec" (UID: "5cf14527-ef89-4011-bad8-4ccd4bc245ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:04:52.867542 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:52.867522    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf14527-ef89-4011-bad8-4ccd4bc245ec-kube-api-access-9wvtm" pod "5cf14527-ef89-4011-bad8-4ccd4bc245ec" (UID: "5cf14527-ef89-4011-bad8-4ccd4bc245ec"). InnerVolumeSpecName "kube-api-access-9wvtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:04:52.867610 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:52.867582    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cf14527-ef89-4011-bad8-4ccd4bc245ec-webhook-cert" pod "5cf14527-ef89-4011-bad8-4ccd4bc245ec" (UID: "5cf14527-ef89-4011-bad8-4ccd4bc245ec"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:04:52.966155 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:52.966113    4660 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cf14527-ef89-4011-bad8-4ccd4bc245ec-webhook-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:52.966155 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:52.966150    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wvtm\" (UniqueName: \"kubernetes.io/projected/5cf14527-ef89-4011-bad8-4ccd4bc245ec-kube-api-access-9wvtm\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:52.966155 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:52.966162    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cf14527-ef89-4011-bad8-4ccd4bc245ec-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:53.642197 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:53.642165    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56949457b6-mx968"
Oct 31 14:04:53.642726 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:53.642191    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56949457b6-mx968" event={"ID":"5cf14527-ef89-4011-bad8-4ccd4bc245ec","Type":"ContainerDied","Data":"0581c4af733602b6b1995043f99ea553eb12522fc60060dc0d8f09b433476637"}
Oct 31 14:04:53.642726 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:53.642267    4660 scope.go:122] "RemoveContainer" containerID="200bf4f0eab1c415e741b153bb49adf1cbd838c9a8c6d25f9af93eb96fd23f61"
Oct 31 14:04:53.668054 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:53.668025    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56949457b6-mx968"]
Oct 31 14:04:53.674701 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:53.674670    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56949457b6-mx968"]
Oct 31 14:04:54.529320 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:54.529238    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cf14527-ef89-4011-bad8-4ccd4bc245ec" path="/var/lib/kubelet/pods/5cf14527-ef89-4011-bad8-4ccd4bc245ec/volumes"
Oct 31 14:04:54.947245 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:54.947204    4660 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 14:04:54.947690 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:54.947276    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 14:04:55.122884 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.122809    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 14:04:55.188210 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.188175    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-podinfo\") pod \"e64a3375-fe5e-426f-bc22-bcaca79b2722\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") "
Oct 31 14:04:55.188210 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.188221    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-token\" (UniqueName: \"kubernetes.io/projected/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-token\") pod \"e64a3375-fe5e-426f-bc22-bcaca79b2722\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") "
Oct 31 14:04:55.188510 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.188269    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-data\") pod \"e64a3375-fe5e-426f-bc22-bcaca79b2722\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") "
Oct 31 14:04:55.188510 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.188342    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-workload-certs\") pod \"e64a3375-fe5e-426f-bc22-bcaca79b2722\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") "
Oct 31 14:04:55.188510 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.188390    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64a3375-fe5e-426f-bc22-bcaca79b2722-kube-api-access-wd2r7\" (UniqueName: \"kubernetes.io/projected/e64a3375-fe5e-426f-bc22-bcaca79b2722-kube-api-access-wd2r7\") pod \"e64a3375-fe5e-426f-bc22-bcaca79b2722\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") "
Oct 31 14:04:55.188510 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.188433    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64a3375-fe5e-426f-bc22-bcaca79b2722-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64a3375-fe5e-426f-bc22-bcaca79b2722-istiod-ca-cert\") pod \"e64a3375-fe5e-426f-bc22-bcaca79b2722\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") "
Oct 31 14:04:55.188510 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.188469    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-envoy\") pod \"e64a3375-fe5e-426f-bc22-bcaca79b2722\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") "
Oct 31 14:04:55.188813 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.188519    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-credential-socket\") pod \"e64a3375-fe5e-426f-bc22-bcaca79b2722\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") "
Oct 31 14:04:55.188813 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.188595    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-workload-socket\") pod \"e64a3375-fe5e-426f-bc22-bcaca79b2722\" (UID: \"e64a3375-fe5e-426f-bc22-bcaca79b2722\") "
Oct 31 14:04:55.188813 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.188672    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-data" pod "e64a3375-fe5e-426f-bc22-bcaca79b2722" (UID: "e64a3375-fe5e-426f-bc22-bcaca79b2722"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:55.188813 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.188730    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-workload-certs" pod "e64a3375-fe5e-426f-bc22-bcaca79b2722" (UID: "e64a3375-fe5e-426f-bc22-bcaca79b2722"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:55.189025 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.188960    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-credential-socket" pod "e64a3375-fe5e-426f-bc22-bcaca79b2722" (UID: "e64a3375-fe5e-426f-bc22-bcaca79b2722"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:55.189025 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.188970    4660 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-data\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.189025 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.188993    4660 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-workload-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.189197 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.189061    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-workload-socket" pod "e64a3375-fe5e-426f-bc22-bcaca79b2722" (UID: "e64a3375-fe5e-426f-bc22-bcaca79b2722"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:55.189197 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.189162    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64a3375-fe5e-426f-bc22-bcaca79b2722-istiod-ca-cert" pod "e64a3375-fe5e-426f-bc22-bcaca79b2722" (UID: "e64a3375-fe5e-426f-bc22-bcaca79b2722"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:04:55.190632 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.190603    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-token" pod "e64a3375-fe5e-426f-bc22-bcaca79b2722" (UID: "e64a3375-fe5e-426f-bc22-bcaca79b2722"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:04:55.191134 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.191102    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64a3375-fe5e-426f-bc22-bcaca79b2722-kube-api-access-wd2r7" pod "e64a3375-fe5e-426f-bc22-bcaca79b2722" (UID: "e64a3375-fe5e-426f-bc22-bcaca79b2722"). InnerVolumeSpecName "kube-api-access-wd2r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:04:55.191250 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.191191    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-envoy" pod "e64a3375-fe5e-426f-bc22-bcaca79b2722" (UID: "e64a3375-fe5e-426f-bc22-bcaca79b2722"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:55.191250 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.191208    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-podinfo" pod "e64a3375-fe5e-426f-bc22-bcaca79b2722" (UID: "e64a3375-fe5e-426f-bc22-bcaca79b2722"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:04:55.289450 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.289416    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd2r7\" (UniqueName: \"kubernetes.io/projected/e64a3375-fe5e-426f-bc22-bcaca79b2722-kube-api-access-wd2r7\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.289450 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.289445    4660 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64a3375-fe5e-426f-bc22-bcaca79b2722-istiod-ca-cert\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.289450 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.289456    4660 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-envoy\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.289695 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.289477    4660 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-credential-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.289695 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.289486    4660 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64a3375-fe5e-426f-bc22-bcaca79b2722-workload-socket\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.289695 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.289496    4660 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-podinfo\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.289695 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.289505    4660 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64a3375-fe5e-426f-bc22-bcaca79b2722-istio-token\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.660238 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.660198    4660 generic.go:356] "Generic (PLEG): container finished" podID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerID="cdf87235268786cf2bc510827622283b3d5f06ad016a2cd81c385b3bc5be62f5" exitCode=0
Oct 31 14:04:55.660442 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.660274    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"
Oct 31 14:04:55.660442 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.660288    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" event={"ID":"e64a3375-fe5e-426f-bc22-bcaca79b2722","Type":"ContainerDied","Data":"cdf87235268786cf2bc510827622283b3d5f06ad016a2cd81c385b3bc5be62f5"}
Oct 31 14:04:55.660442 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.660365    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv" event={"ID":"e64a3375-fe5e-426f-bc22-bcaca79b2722","Type":"ContainerDied","Data":"06ed7abaf91f20ae7b2d19d4ed6a9bcd85e0dd362f5f46ea374aada3f34e2ae6"}
Oct 31 14:04:55.660442 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.660390    4660 scope.go:122] "RemoveContainer" containerID="cdf87235268786cf2bc510827622283b3d5f06ad016a2cd81c385b3bc5be62f5"
Oct 31 14:04:55.683912 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.683884    4660 scope.go:122] "RemoveContainer" containerID="cdf87235268786cf2bc510827622283b3d5f06ad016a2cd81c385b3bc5be62f5"
Oct 31 14:04:55.684338 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:04:55.684265    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf87235268786cf2bc510827622283b3d5f06ad016a2cd81c385b3bc5be62f5\": container with ID starting with cdf87235268786cf2bc510827622283b3d5f06ad016a2cd81c385b3bc5be62f5 not found: ID does not exist" containerID="cdf87235268786cf2bc510827622283b3d5f06ad016a2cd81c385b3bc5be62f5"
Oct 31 14:04:55.684885 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.684862    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf87235268786cf2bc510827622283b3d5f06ad016a2cd81c385b3bc5be62f5"} err="failed to get container status \"cdf87235268786cf2bc510827622283b3d5f06ad016a2cd81c385b3bc5be62f5\": rpc error: code = NotFound desc = could not find container \"cdf87235268786cf2bc510827622283b3d5f06ad016a2cd81c385b3bc5be62f5\": container with ID starting with cdf87235268786cf2bc510827622283b3d5f06ad016a2cd81c385b3bc5be62f5 not found: ID does not exist"
Oct 31 14:04:55.698100 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.698062    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"]
Oct 31 14:04:55.700403 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:55.700382    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-l5cvv"]
Oct 31 14:04:56.526254 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:04:56.526216    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" path="/var/lib/kubelet/pods/e64a3375-fe5e-426f-bc22-bcaca79b2722/volumes"
Oct 31 14:05:07.029160 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:07.029118    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"]
Oct 31 14:05:07.033495 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:07.033465    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273ce012-8s4w4"]
Oct 31 14:05:08.526854 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:08.526807    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0" path="/var/lib/kubelet/pods/dd98cce8-cedf-4e5c-957d-7f9c79d7f6f0/volumes"
Oct 31 14:05:13.834237 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834195    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"]
Oct 31 14:05:13.834824 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834799    4660 cpu_manager.go:414] "containerMap: removing container" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy"
Oct 31 14:05:13.834901 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834827    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy"
Oct 31 14:05:13.834901 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834852    4660 cpu_manager.go:414] "containerMap: removing container" podUID="21f3684e-26dd-493c-af92-89e1f8b2b18a" containerName="maas-api"
Oct 31 14:05:13.834901 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834860    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21f3684e-26dd-493c-af92-89e1f8b2b18a" containerName="maas-api"
Oct 31 14:05:13.834901 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834876    4660 cpu_manager.go:414] "containerMap: removing container" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" containerName="istio-proxy"
Oct 31 14:05:13.834901 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834881    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" containerName="istio-proxy"
Oct 31 14:05:13.834901 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834889    4660 cpu_manager.go:414] "containerMap: removing container" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" containerName="istio-proxy"
Oct 31 14:05:13.834901 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834894    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" containerName="istio-proxy"
Oct 31 14:05:13.834901 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834902    4660 cpu_manager.go:414] "containerMap: removing container" podUID="5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834909    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834920    4660 cpu_manager.go:414] "containerMap: removing container" podUID="5839b96b-4f18-4d9a-93df-42da4ba9e666" containerName="cleanup"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834928    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5839b96b-4f18-4d9a-93df-42da4ba9e666" containerName="cleanup"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834942    4660 cpu_manager.go:414] "containerMap: removing container" podUID="72fd0bc3-694c-44cc-8309-f59b16ab2561" containerName="cleanup"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834947    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72fd0bc3-694c-44cc-8309-f59b16ab2561" containerName="cleanup"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834954    4660 cpu_manager.go:414] "containerMap: removing container" podUID="3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e" containerName="maas-api"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834959    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e" containerName="maas-api"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834969    4660 cpu_manager.go:414] "containerMap: removing container" podUID="533c7ae9-3ae7-4cd4-a38f-98c35110b250" containerName="maas-api"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834976    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533c7ae9-3ae7-4cd4-a38f-98c35110b250" containerName="maas-api"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834984    4660 cpu_manager.go:414] "containerMap: removing container" podUID="ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834988    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.834996    4660 cpu_manager.go:414] "containerMap: removing container" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835000    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835009    4660 cpu_manager.go:414] "containerMap: removing container" podUID="344707b1-2bf5-4481-8522-afe6d996c26c" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835013    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="344707b1-2bf5-4481-8522-afe6d996c26c" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835020    4660 cpu_manager.go:414] "containerMap: removing container" podUID="de5ecce4-e294-4623-b83d-9ba9a5e74bca" containerName="maas-api"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835024    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5ecce4-e294-4623-b83d-9ba9a5e74bca" containerName="maas-api"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835031    4660 cpu_manager.go:414] "containerMap: removing container" podUID="a94c907a-8bdd-4b73-86ee-3c132831df1f" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835035    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94c907a-8bdd-4b73-86ee-3c132831df1f" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835042    4660 cpu_manager.go:414] "containerMap: removing container" podUID="146bb004-162d-4777-92b9-08b3120df4d1" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835046    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146bb004-162d-4777-92b9-08b3120df4d1" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835054    4660 cpu_manager.go:414] "containerMap: removing container" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835058    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835067    4660 cpu_manager.go:414] "containerMap: removing container" podUID="84174b11-1866-4f83-ad10-84a6288b53dd" containerName="payload-processing"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835073    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84174b11-1866-4f83-ad10-84a6288b53dd" containerName="payload-processing"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835079    4660 cpu_manager.go:414] "containerMap: removing container" podUID="e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9" containerName="maas-api"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835084    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9" containerName="maas-api"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835096    4660 cpu_manager.go:414] "containerMap: removing container" podUID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835102    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835109    4660 cpu_manager.go:414] "containerMap: removing container" podUID="f82f37eb-0856-4af8-857b-b855316ee6b0" containerName="maas-api"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835116    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82f37eb-0856-4af8-857b-b855316ee6b0" containerName="maas-api"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835125    4660 cpu_manager.go:414] "containerMap: removing container" podUID="44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835133    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835148    4660 cpu_manager.go:414] "containerMap: removing container" podUID="8e133f4e-385e-46b4-ab50-db03976321e0" containerName="maas-api"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835155    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e133f4e-385e-46b4-ab50-db03976321e0" containerName="maas-api"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835164    4660 cpu_manager.go:414] "containerMap: removing container" podUID="cc3f6cec-989a-4415-8328-389d1610947f" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835171    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc3f6cec-989a-4415-8328-389d1610947f" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835181    4660 cpu_manager.go:414] "containerMap: removing container" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835188    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835197    4660 cpu_manager.go:414] "containerMap: removing container" podUID="1b061377-3c76-4eb8-aa22-04425a306fe6" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835204    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b061377-3c76-4eb8-aa22-04425a306fe6" containerName="istio-proxy"
Oct 31 14:05:13.835194 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835217    4660 cpu_manager.go:414] "containerMap: removing container" podUID="5cf14527-ef89-4011-bad8-4ccd4bc245ec" containerName="manager"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835223    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf14527-ef89-4011-bad8-4ccd4bc245ec" containerName="manager"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835234    4660 cpu_manager.go:414] "containerMap: removing container" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835240    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835247    4660 cpu_manager.go:414] "containerMap: removing container" podUID="f696d368-e075-482d-8e6f-da424076afa0" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835254    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f696d368-e075-482d-8e6f-da424076afa0" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835265    4660 cpu_manager.go:414] "containerMap: removing container" podUID="04449154-f556-4576-b8c1-1a4ec39c960a" containerName="maas-api"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835272    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04449154-f556-4576-b8c1-1a4ec39c960a" containerName="maas-api"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835281    4660 cpu_manager.go:414] "containerMap: removing container" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835287    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835311    4660 cpu_manager.go:414] "containerMap: removing container" podUID="f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835318    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835327    4660 cpu_manager.go:414] "containerMap: removing container" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835333    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835346    4660 cpu_manager.go:414] "containerMap: removing container" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835352    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835366    4660 cpu_manager.go:414] "containerMap: removing container" podUID="6a36003d-cda6-447e-bb9f-95f13e32c276" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835373    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a36003d-cda6-447e-bb9f-95f13e32c276" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835383    4660 cpu_manager.go:414] "containerMap: removing container" podUID="fb081f1b-a184-4656-a834-240605f9693b" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835390    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb081f1b-a184-4656-a834-240605f9693b" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835401    4660 cpu_manager.go:414] "containerMap: removing container" podUID="5db28b27-5dcf-4468-8ebc-951e8f52613f" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835408    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db28b27-5dcf-4468-8ebc-951e8f52613f" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835419    4660 cpu_manager.go:414] "containerMap: removing container" podUID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835426    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835525    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f143333-68f6-413a-a882-71ce428ba0b1" containerName="cleanup"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835538    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835547    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="be5b97ff-dbf0-4a26-835f-702598486d61" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835552    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="5549d7d1-08ba-419c-8cf1-9c2b29ceaec3" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835559    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ccf2335-bed6-41a0-8c33-9ddb0b6f7f9e" containerName="maas-api"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835566    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="533c7ae9-3ae7-4cd4-a38f-98c35110b250" containerName="maas-api"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835571    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed888973-2d46-4359-8d0e-dd2e7f6fc0b2" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835577    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="28afbd7d-8b18-44e6-8023-8a1019197ceb" containerName="main"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835585    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="344707b1-2bf5-4481-8522-afe6d996c26c" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835594    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5ecce4-e294-4623-b83d-9ba9a5e74bca" containerName="maas-api"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835602    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="a94c907a-8bdd-4b73-86ee-3c132831df1f" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835608    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="146bb004-162d-4777-92b9-08b3120df4d1" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835614    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64a3375-fe5e-426f-bc22-bcaca79b2722" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835622    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="84174b11-1866-4f83-ad10-84a6288b53dd" containerName="payload-processing"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835628    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6db6841-fcb1-48f5-bbc0-1fe2bb2727e9" containerName="maas-api"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835634    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835639    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82f37eb-0856-4af8-857b-b855316ee6b0" containerName="maas-api"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835644    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="44fe7652-1fd9-470e-b8f9-a3cbcc061a8c" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835650    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e133f4e-385e-46b4-ab50-db03976321e0" containerName="maas-api"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835655    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc3f6cec-989a-4415-8328-389d1610947f" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835661    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68b0a7c-08d8-42e2-8b56-f81b58386500" containerName="main"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835669    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b061377-3c76-4eb8-aa22-04425a306fe6" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835677    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf14527-ef89-4011-bad8-4ccd4bc245ec" containerName="manager"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835682    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce80f7f-51e8-46f3-add6-37bbb3a0adba" containerName="main"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835687    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="f696d368-e075-482d-8e6f-da424076afa0" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835692    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="04449154-f556-4576-b8c1-1a4ec39c960a" containerName="maas-api"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835697    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd80067-027a-40ab-be2f-63e7597a72f3" containerName="main"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835702    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2dc5c7e-d033-4f1d-92e9-b8c57128d7a8" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835708    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4bb048a-1211-4eea-b483-f1c22a9da6a5" containerName="main"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835714    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="38522f51-cc0e-4026-a6f3-a9be0b429561" containerName="main"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835722    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="c687b3e0-c154-4c13-9f6e-0e05e7db3ac5" containerName="cleanup"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835732    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a36003d-cda6-447e-bb9f-95f13e32c276" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835740    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb081f1b-a184-4656-a834-240605f9693b" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835748    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db28b27-5dcf-4468-8ebc-951e8f52613f" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835755    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa19833b-88b6-453e-81c8-4ce97c30669e" containerName="istio-proxy"
Oct 31 14:05:13.837167 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.835763    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="21f3684e-26dd-493c-af92-89e1f8b2b18a" containerName="maas-api"
Oct 31 14:05:13.840800 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.840780    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:13.853946 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.853914    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"]
Oct 31 14:05:13.970569 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.970527    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/541446ef-8e36-4be7-9a16-b4005f6c250a-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:13.970759 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.970583    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/541446ef-8e36-4be7-9a16-b4005f6c250a-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:13.970759 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.970668    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/541446ef-8e36-4be7-9a16-b4005f6c250a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:13.970759 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.970736    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/541446ef-8e36-4be7-9a16-b4005f6c250a-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:13.970897 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.970773    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/541446ef-8e36-4be7-9a16-b4005f6c250a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:13.970897 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.970824    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/541446ef-8e36-4be7-9a16-b4005f6c250a-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:13.970897 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.970852    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/541446ef-8e36-4be7-9a16-b4005f6c250a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:13.970999 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.970905    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7fcf\" (UniqueName: \"kubernetes.io/projected/541446ef-8e36-4be7-9a16-b4005f6c250a-kube-api-access-x7fcf\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:13.970999 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:13.970927    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/541446ef-8e36-4be7-9a16-b4005f6c250a-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.071480 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.071438    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/541446ef-8e36-4be7-9a16-b4005f6c250a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.071480 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.071480    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/541446ef-8e36-4be7-9a16-b4005f6c250a-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.071774 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.071513    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/541446ef-8e36-4be7-9a16-b4005f6c250a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.071774 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.071564    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7fcf\" (UniqueName: \"kubernetes.io/projected/541446ef-8e36-4be7-9a16-b4005f6c250a-kube-api-access-x7fcf\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.071774 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.071590    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/541446ef-8e36-4be7-9a16-b4005f6c250a-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.071774 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.071740    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/541446ef-8e36-4be7-9a16-b4005f6c250a-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.072013 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.071832    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/541446ef-8e36-4be7-9a16-b4005f6c250a-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.072013 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.071874    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/541446ef-8e36-4be7-9a16-b4005f6c250a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.072013 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.071908    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/541446ef-8e36-4be7-9a16-b4005f6c250a-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.072013 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.071939    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/541446ef-8e36-4be7-9a16-b4005f6c250a-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.072013 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.071960    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/541446ef-8e36-4be7-9a16-b4005f6c250a-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.072348 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.072323    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/541446ef-8e36-4be7-9a16-b4005f6c250a-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.072595 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.072570    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/541446ef-8e36-4be7-9a16-b4005f6c250a-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.072674 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.072627    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/541446ef-8e36-4be7-9a16-b4005f6c250a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.074392 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.074361    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/541446ef-8e36-4be7-9a16-b4005f6c250a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.074642 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.074623    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/541446ef-8e36-4be7-9a16-b4005f6c250a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.086502 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.086419    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7fcf\" (UniqueName: \"kubernetes.io/projected/541446ef-8e36-4be7-9a16-b4005f6c250a-kube-api-access-x7fcf\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.086724 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.086698    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/541446ef-8e36-4be7-9a16-b4005f6c250a-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-5z724\" (UID: \"541446ef-8e36-4be7-9a16-b4005f6c250a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.157810 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.157768    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:14.450069 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.450040    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"]
Oct 31 14:05:14.453333 ip-10-0-137-77 kubenswrapper[4660]: W1031 14:05:14.453305    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541446ef_8e36_4be7_9a16_b4005f6c250a.slice/crio-3060d1ad35e76af3433138281ddfc265a9513e68cb9f24e2ad544f50f14994c3 WatchSource:0}: Error finding container 3060d1ad35e76af3433138281ddfc265a9513e68cb9f24e2ad544f50f14994c3: Status 404 returned error can't find the container with id 3060d1ad35e76af3433138281ddfc265a9513e68cb9f24e2ad544f50f14994c3
Oct 31 14:05:14.455532 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.455499    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 14:05:14.455628 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.455570    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 14:05:14.455628 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.455600    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 14:05:14.771028 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.770924    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724" event={"ID":"541446ef-8e36-4be7-9a16-b4005f6c250a","Type":"ContainerStarted","Data":"7b371e5dfd9976f359b834706871efb13acd3c80fae9e260ea2f48e3fafae929"}
Oct 31 14:05:14.771028 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.770997    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724" event={"ID":"541446ef-8e36-4be7-9a16-b4005f6c250a","Type":"ContainerStarted","Data":"3060d1ad35e76af3433138281ddfc265a9513e68cb9f24e2ad544f50f14994c3"}
Oct 31 14:05:14.798662 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:14.798595    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724" podStartSLOduration=1.798575467 podStartE2EDuration="1.798575467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:05:14.794021198 +0000 UTC m=+3714.412680362" watchObservedRunningTime="2026-10-31 14:05:14.798575467 +0000 UTC m=+3714.417234632"
Oct 31 14:05:15.158461 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:15.158423    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:15.160118 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:15.160090    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5z724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 14:05:15.160237 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:15.160144    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724" podUID="541446ef-8e36-4be7-9a16-b4005f6c250a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 14:05:16.158581 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:16.158536    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5z724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 14:05:16.158966 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:16.158599    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724" podUID="541446ef-8e36-4be7-9a16-b4005f6c250a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 14:05:17.158456 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:17.158417    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5z724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 14:05:17.158653 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:17.158489    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724" podUID="541446ef-8e36-4be7-9a16-b4005f6c250a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 14:05:18.045202 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:18.045162    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"]
Oct 31 14:05:18.052055 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:18.052025    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c5f7a1-k2qvb"]
Oct 31 14:05:18.158473 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:18.158439    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5z724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 14:05:18.158640 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:18.158491    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724" podUID="541446ef-8e36-4be7-9a16-b4005f6c250a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 14:05:18.531222 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:18.531186    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1281827e-5601-49c6-8822-a48e736b47ee" path="/var/lib/kubelet/pods/1281827e-5601-49c6-8822-a48e736b47ee/volumes"
Oct 31 14:05:19.158410 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:19.158375    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5z724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 14:05:19.158626 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:19.158435    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724" podUID="541446ef-8e36-4be7-9a16-b4005f6c250a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 14:05:20.159189 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:20.159150    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5z724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 14:05:20.159628 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:20.159207    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724" podUID="541446ef-8e36-4be7-9a16-b4005f6c250a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 14:05:20.733795 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:05:20.733754    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:05:21.158635 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:21.158600    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5z724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 14:05:21.158884 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:21.158656    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724" podUID="541446ef-8e36-4be7-9a16-b4005f6c250a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 14:05:23.158611 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:23.158566    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5z724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 14:05:23.159024 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:23.158631    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724" podUID="541446ef-8e36-4be7-9a16-b4005f6c250a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 14:05:24.201440 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:24.201402    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-5z724 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 14:05:24.201952 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:24.201458    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724" podUID="541446ef-8e36-4be7-9a16-b4005f6c250a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 14:05:24.716861 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:24.716824    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:24.717230 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:24.717204    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:24.719602 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:24.719581    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-5z724"
Oct 31 14:05:28.847885 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:28.847850    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"]
Oct 31 14:05:28.848347 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:28.848332    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71b679c-5dff-4c19-bef4-a4bf19093b34" containerName="istio-proxy"
Oct 31 14:05:28.852431 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:28.852408    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:28.868149 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:28.868118    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"]
Oct 31 14:05:28.929922 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:28.929885    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:28.929922 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:28.929924    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9fz2\" (UniqueName: \"kubernetes.io/projected/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-kube-api-access-d9fz2\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:28.930155 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:28.929952    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:28.930155 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:28.929971    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:28.930269 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:28.930153    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:28.930350 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:28.930280    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:28.930350 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:28.930344    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:28.930441 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:28.930382    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:28.930528 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:28.930503    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.031238 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.031199    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.031476 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.031261    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.031476 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.031285    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9fz2\" (UniqueName: \"kubernetes.io/projected/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-kube-api-access-d9fz2\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.031476 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.031334    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.031476 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.031350    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.031476 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.031374    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.031476 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.031437    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.031476 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.031467    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.031925 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.031504    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.031925 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.031736    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.031925 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.031820    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.031925 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.031847    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.032062 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.031998    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.032062 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.032052    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.033793 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.033767    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.034016 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.033997    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.047271 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.047248    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.047679 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.047655    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9fz2\" (UniqueName: \"kubernetes.io/projected/876ecf6e-3c48-4f96-8eaa-d7d5196be2ca-kube-api-access-d9fz2\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-gwl2w\" (UID: \"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.165613 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.165582    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:29.455367 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.455338    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"]
Oct 31 14:05:29.457035 ip-10-0-137-77 kubenswrapper[4660]: W1031 14:05:29.457011    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod876ecf6e_3c48_4f96_8eaa_d7d5196be2ca.slice/crio-b550378ca9ecf3651b9708cc07c78922547a338fe15e0b47c79ffbd0859b3405 WatchSource:0}: Error finding container b550378ca9ecf3651b9708cc07c78922547a338fe15e0b47c79ffbd0859b3405: Status 404 returned error can't find the container with id b550378ca9ecf3651b9708cc07c78922547a338fe15e0b47c79ffbd0859b3405
Oct 31 14:05:29.459219 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.459189    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 14:05:29.459282 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.459257    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 14:05:29.459353 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.459287    4660 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 14:05:29.869882 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.869846    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w" event={"ID":"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca","Type":"ContainerStarted","Data":"0ff298b4ed2035dcd067d9930dcc8e1dac42461f8706c50a01bbf0e6e312c21e"}
Oct 31 14:05:29.870348 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.869910    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w" event={"ID":"876ecf6e-3c48-4f96-8eaa-d7d5196be2ca","Type":"ContainerStarted","Data":"b550378ca9ecf3651b9708cc07c78922547a338fe15e0b47c79ffbd0859b3405"}
Oct 31 14:05:29.895561 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:29.895513    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w" podStartSLOduration=1.895499297 podStartE2EDuration="1.895499297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:05:29.893903473 +0000 UTC m=+3729.512562637" watchObservedRunningTime="2026-10-31 14:05:29.895499297 +0000 UTC m=+3729.514158460"
Oct 31 14:05:30.166140 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:30.166025    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:30.167549 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:30.167525    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 31 14:05:30.167619 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:30.167576    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w" podUID="876ecf6e-3c48-4f96-8eaa-d7d5196be2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 31 14:05:31.166481 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:31.166442    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 31 14:05:31.166894 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:31.166500    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w" podUID="876ecf6e-3c48-4f96-8eaa-d7d5196be2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 31 14:05:32.166388 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:32.166354    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 31 14:05:32.166572 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:32.166405    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w" podUID="876ecf6e-3c48-4f96-8eaa-d7d5196be2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 31 14:05:33.166862 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:33.166823    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 31 14:05:33.167276 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:33.166885    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w" podUID="876ecf6e-3c48-4f96-8eaa-d7d5196be2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 31 14:05:34.166013 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:34.165970    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 31 14:05:34.166388 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:34.166029    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w" podUID="876ecf6e-3c48-4f96-8eaa-d7d5196be2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 31 14:05:35.166195 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:35.166153    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 31 14:05:35.166656 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:35.166210    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w" podUID="876ecf6e-3c48-4f96-8eaa-d7d5196be2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 31 14:05:36.166885 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:36.166848    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 31 14:05:36.167334 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:36.166932    4660 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w" podUID="876ecf6e-3c48-4f96-8eaa-d7d5196be2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 31 14:05:37.654909 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:37.654874    4660 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:37.655457 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:37.655316    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:37.656128 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:05:37.656108    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w"
Oct 31 14:05:50.731171 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:05:50.731131    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:06:20.872319 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:06:20.872078    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:06:25.914785 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:06:25.914750    4660 scope.go:122] "RemoveContainer" containerID="3b4a1e31aefef15b9af9606aece14391d9dd0dcc7137b173081618063795ebba"
Oct 31 14:06:25.923819 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:06:25.923799    4660 scope.go:122] "RemoveContainer" containerID="8e5ea0b4bba94006346ff6218b974439ea00d0b8a9cc25e8eb2c250f4bfaf50b"
Oct 31 14:06:50.732110 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:06:50.732065    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:06:53.700494 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:06:53.700453    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 14:06:53.700957 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:06:53.700519    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w" podUID="876ecf6e-3c48-4f96-8eaa-d7d5196be2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 14:07:18.111545 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:07:18.111420    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 14:07:18.111545 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:07:18.111516    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9" podUID="ab9e59d3-2ac8-4381-98fa-22d3f25d682b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 14:07:20.731531 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:07:20.731488    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:07:50.795443 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:07:50.795390    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:08:20.732031 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:08:20.731990    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:08:21.430094 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:08:21.430062    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 14:08:21.435187 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:08:21.435160    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 14:08:21.436275 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:08:21.436217    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 14:08:21.441250 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:08:21.441230    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 14:08:21.444901 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:08:21.444878    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 14:08:21.449897 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:08:21.449878    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 14:08:50.731872 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:08:50.731833    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:09:20.732959 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:09:20.732915    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:09:23.698517 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:09:23.698466    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 14:09:23.698932 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:09:23.698545    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-gwl2w" podUID="876ecf6e-3c48-4f96-8eaa-d7d5196be2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 14:09:50.737489 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:09:50.737441    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:10:20.747477 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:10:20.747381    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:10:50.732831 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:10:50.732789    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:11:20.734563 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:11:20.734521    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:11:50.796000 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:11:50.795946    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:11:59.591918 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:11:59.591876    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"]
Oct 31 14:11:59.592945 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:11:59.592075    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x" podUID="479e3854-ed33-4732-a369-d7d367ba245d" containerName="manager" containerID="cri-o://b9a6560dd97bdb7b86e40f6d54285d619c183fd78d34e7e0310ef874af3e1755" gracePeriod=10
Oct 31 14:11:59.856197 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:11:59.856134    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"
Oct 31 14:11:59.868329 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:11:59.868278    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479e3854-ed33-4732-a369-d7d367ba245d-kube-api-access-p9w6f\" (UniqueName: \"kubernetes.io/projected/479e3854-ed33-4732-a369-d7d367ba245d-kube-api-access-p9w6f\") pod \"479e3854-ed33-4732-a369-d7d367ba245d\" (UID: \"479e3854-ed33-4732-a369-d7d367ba245d\") "
Oct 31 14:11:59.868486 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:11:59.868359    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479e3854-ed33-4732-a369-d7d367ba245d-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/479e3854-ed33-4732-a369-d7d367ba245d-extensions-socket-volume\") pod \"479e3854-ed33-4732-a369-d7d367ba245d\" (UID: \"479e3854-ed33-4732-a369-d7d367ba245d\") "
Oct 31 14:11:59.868853 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:11:59.868825    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479e3854-ed33-4732-a369-d7d367ba245d-extensions-socket-volume" pod "479e3854-ed33-4732-a369-d7d367ba245d" (UID: "479e3854-ed33-4732-a369-d7d367ba245d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:11:59.870407 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:11:59.870385    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479e3854-ed33-4732-a369-d7d367ba245d-kube-api-access-p9w6f" pod "479e3854-ed33-4732-a369-d7d367ba245d" (UID: "479e3854-ed33-4732-a369-d7d367ba245d"). InnerVolumeSpecName "kube-api-access-p9w6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:11:59.969279 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:11:59.969238    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9w6f\" (UniqueName: \"kubernetes.io/projected/479e3854-ed33-4732-a369-d7d367ba245d-kube-api-access-p9w6f\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:59.969279 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:11:59.969271    4660 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/479e3854-ed33-4732-a369-d7d367ba245d-extensions-socket-volume\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:12:00.566453 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:12:00.566418    4660 generic.go:356] "Generic (PLEG): container finished" podID="479e3854-ed33-4732-a369-d7d367ba245d" containerID="b9a6560dd97bdb7b86e40f6d54285d619c183fd78d34e7e0310ef874af3e1755" exitCode=0
Oct 31 14:12:00.566629 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:12:00.566466    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"
Oct 31 14:12:00.566629 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:12:00.566492    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x" event={"ID":"479e3854-ed33-4732-a369-d7d367ba245d","Type":"ContainerDied","Data":"b9a6560dd97bdb7b86e40f6d54285d619c183fd78d34e7e0310ef874af3e1755"}
Oct 31 14:12:00.566629 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:12:00.566546    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x" event={"ID":"479e3854-ed33-4732-a369-d7d367ba245d","Type":"ContainerDied","Data":"16e8793ab85bc57d81a8e884bf844907fe4da128985947e487500ca33e4b3a70"}
Oct 31 14:12:00.566629 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:12:00.566569    4660 scope.go:122] "RemoveContainer" containerID="b9a6560dd97bdb7b86e40f6d54285d619c183fd78d34e7e0310ef874af3e1755"
Oct 31 14:12:00.579603 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:12:00.579578    4660 scope.go:122] "RemoveContainer" containerID="b9a6560dd97bdb7b86e40f6d54285d619c183fd78d34e7e0310ef874af3e1755"
Oct 31 14:12:00.579967 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:12:00.579943    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a6560dd97bdb7b86e40f6d54285d619c183fd78d34e7e0310ef874af3e1755\": container with ID starting with b9a6560dd97bdb7b86e40f6d54285d619c183fd78d34e7e0310ef874af3e1755 not found: ID does not exist" containerID="b9a6560dd97bdb7b86e40f6d54285d619c183fd78d34e7e0310ef874af3e1755"
Oct 31 14:12:00.580068 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:12:00.579975    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a6560dd97bdb7b86e40f6d54285d619c183fd78d34e7e0310ef874af3e1755"} err="failed to get container status \"b9a6560dd97bdb7b86e40f6d54285d619c183fd78d34e7e0310ef874af3e1755\": rpc error: code = NotFound desc = could not find container \"b9a6560dd97bdb7b86e40f6d54285d619c183fd78d34e7e0310ef874af3e1755\": container with ID starting with b9a6560dd97bdb7b86e40f6d54285d619c183fd78d34e7e0310ef874af3e1755 not found: ID does not exist"
Oct 31 14:12:00.591839 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:12:00.591807    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"]
Oct 31 14:12:00.596849 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:12:00.596824    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9xc4x"]
Oct 31 14:12:02.526162 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:12:02.526124    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479e3854-ed33-4732-a369-d7d367ba245d" path="/var/lib/kubelet/pods/479e3854-ed33-4732-a369-d7d367ba245d/volumes"
Oct 31 14:12:20.731514 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:12:20.731473    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:12:50.732108 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:12:50.732062    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:13:20.730851 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:13:20.730766    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:13:21.475564 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:13:21.475471    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 14:13:21.479806 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:13:21.479783    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 14:13:21.481271 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:13:21.481250    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 14:13:21.485448 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:13:21.485423    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 14:13:21.489734 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:13:21.489711    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 14:13:21.494104 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:13:21.494082    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 14:13:50.730932 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:13:50.730884    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:14:20.731881 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:14:20.731842    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:14:50.731896 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:14:50.731857    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:15:20.731632 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:15:20.731591    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:15:33.107530 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:15:33.107463    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 14:15:33.108114 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:15:33.107558    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9" podUID="ab9e59d3-2ac8-4381-98fa-22d3f25d682b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 14:15:50.732832 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:15:50.732789    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:16:20.733463 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:16:20.733371    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:16:50.731255 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:16:50.731201    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:17:06.622967 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:06.622931    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"]
Oct 31 14:17:06.623561 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:06.623141    4660 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft" podUID="6966c7ae-5ec8-4d5a-9aba-be497296c5f5" containerName="maas-api" containerID="cri-o://e0aa92963ffcde300e2a9394bc5b4a2c1ff4f58a46087e00fa44f3fe13238da2" gracePeriod=30
Oct 31 14:17:07.881267 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:07.881237    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 14:17:07.885953 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:07.885923    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-maas-api-tls\") pod \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\" (UID: \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\") "
Oct 31 14:17:07.886080 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:07.885969    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-openshift-service-ca\") pod \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\" (UID: \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\") "
Oct 31 14:17:07.886080 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:07.886009    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-kube-api-access-mnhqj\" (UniqueName: \"kubernetes.io/projected/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-kube-api-access-mnhqj\") pod \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\" (UID: \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\") "
Oct 31 14:17:07.886412 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:07.886391    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-openshift-service-ca" pod "6966c7ae-5ec8-4d5a-9aba-be497296c5f5" (UID: "6966c7ae-5ec8-4d5a-9aba-be497296c5f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:17:07.888072 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:07.888045    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-maas-api-tls" pod "6966c7ae-5ec8-4d5a-9aba-be497296c5f5" (UID: "6966c7ae-5ec8-4d5a-9aba-be497296c5f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:17:07.888157 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:07.888108    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-kube-api-access-mnhqj" pod "6966c7ae-5ec8-4d5a-9aba-be497296c5f5" (UID: "6966c7ae-5ec8-4d5a-9aba-be497296c5f5"). InnerVolumeSpecName "kube-api-access-mnhqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:17:07.986720 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:07.986621    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-metrics-certs\") pod \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\" (UID: \"6966c7ae-5ec8-4d5a-9aba-be497296c5f5\") "
Oct 31 14:17:07.986951 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:07.986931    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-maas-api-tls\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:07.987010 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:07.986960    4660 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-openshift-service-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:07.987010 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:07.986975    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnhqj\" (UniqueName: \"kubernetes.io/projected/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-kube-api-access-mnhqj\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:07.988778 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:07.988754    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-metrics-certs" pod "6966c7ae-5ec8-4d5a-9aba-be497296c5f5" (UID: "6966c7ae-5ec8-4d5a-9aba-be497296c5f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:17:08.087538 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:08.087483    4660 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6966c7ae-5ec8-4d5a-9aba-be497296c5f5-metrics-certs\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:08.497556 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:08.497516    4660 generic.go:356] "Generic (PLEG): container finished" podID="6966c7ae-5ec8-4d5a-9aba-be497296c5f5" containerID="e0aa92963ffcde300e2a9394bc5b4a2c1ff4f58a46087e00fa44f3fe13238da2" exitCode=0
Oct 31 14:17:08.497765 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:08.497569    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"
Oct 31 14:17:08.497765 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:08.497601    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft" event={"ID":"6966c7ae-5ec8-4d5a-9aba-be497296c5f5","Type":"ContainerDied","Data":"e0aa92963ffcde300e2a9394bc5b4a2c1ff4f58a46087e00fa44f3fe13238da2"}
Oct 31 14:17:08.497765 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:08.497664    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft" event={"ID":"6966c7ae-5ec8-4d5a-9aba-be497296c5f5","Type":"ContainerDied","Data":"06c8d66bf237dff786733f1e2fab404b06ccb6c0a760e9eff287be44720ce151"}
Oct 31 14:17:08.497765 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:08.497693    4660 scope.go:122] "RemoveContainer" containerID="e0aa92963ffcde300e2a9394bc5b4a2c1ff4f58a46087e00fa44f3fe13238da2"
Oct 31 14:17:08.508238 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:08.508213    4660 scope.go:122] "RemoveContainer" containerID="e0aa92963ffcde300e2a9394bc5b4a2c1ff4f58a46087e00fa44f3fe13238da2"
Oct 31 14:17:08.508570 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:17:08.508550    4660 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0aa92963ffcde300e2a9394bc5b4a2c1ff4f58a46087e00fa44f3fe13238da2\": container with ID starting with e0aa92963ffcde300e2a9394bc5b4a2c1ff4f58a46087e00fa44f3fe13238da2 not found: ID does not exist" containerID="e0aa92963ffcde300e2a9394bc5b4a2c1ff4f58a46087e00fa44f3fe13238da2"
Oct 31 14:17:08.508640 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:08.508580    4660 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0aa92963ffcde300e2a9394bc5b4a2c1ff4f58a46087e00fa44f3fe13238da2"} err="failed to get container status \"e0aa92963ffcde300e2a9394bc5b4a2c1ff4f58a46087e00fa44f3fe13238da2\": rpc error: code = NotFound desc = could not find container \"e0aa92963ffcde300e2a9394bc5b4a2c1ff4f58a46087e00fa44f3fe13238da2\": container with ID starting with e0aa92963ffcde300e2a9394bc5b4a2c1ff4f58a46087e00fa44f3fe13238da2 not found: ID does not exist"
Oct 31 14:17:08.527523 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:08.527496    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"]
Oct 31 14:17:08.527659 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:08.527530    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d475f7bf9-9cjft"]
Oct 31 14:17:10.526184 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:10.526138    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6966c7ae-5ec8-4d5a-9aba-be497296c5f5" path="/var/lib/kubelet/pods/6966c7ae-5ec8-4d5a-9aba-be497296c5f5/volumes"
Oct 31 14:17:17.184975 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.184939    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"]
Oct 31 14:17:17.185496 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.185355    4660 cpu_manager.go:414] "containerMap: removing container" podUID="479e3854-ed33-4732-a369-d7d367ba245d" containerName="manager"
Oct 31 14:17:17.185496 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.185367    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479e3854-ed33-4732-a369-d7d367ba245d" containerName="manager"
Oct 31 14:17:17.185496 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.185386    4660 cpu_manager.go:414] "containerMap: removing container" podUID="6966c7ae-5ec8-4d5a-9aba-be497296c5f5" containerName="maas-api"
Oct 31 14:17:17.185496 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.185392    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6966c7ae-5ec8-4d5a-9aba-be497296c5f5" containerName="maas-api"
Oct 31 14:17:17.185496 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.185471    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="6966c7ae-5ec8-4d5a-9aba-be497296c5f5" containerName="maas-api"
Oct 31 14:17:17.185496 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.185484    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="479e3854-ed33-4732-a369-d7d367ba245d" containerName="manager"
Oct 31 14:17:17.189848 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.189827    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"
Oct 31 14:17:17.195779 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.195752    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 14:17:17.195923 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.195863    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 14:17:17.195923 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.195861    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:17:17.196073 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.196030    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:17:17.198743 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.198720    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"]
Oct 31 14:17:17.263871 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.263835    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3eb393f-68f4-408c-a739-538d41b5b86c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1008181-p5r5g\" (UID: \"b3eb393f-68f4-408c-a739-538d41b5b86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"
Oct 31 14:17:17.264062 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.263943    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3eb393f-68f4-408c-a739-538d41b5b86c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1008181-p5r5g\" (UID: \"b3eb393f-68f4-408c-a739-538d41b5b86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"
Oct 31 14:17:17.264062 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.263991    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcvxx\" (UniqueName: \"kubernetes.io/projected/b3eb393f-68f4-408c-a739-538d41b5b86c-kube-api-access-qcvxx\") pod \"maas-api-revoke-keys-e2e-ait-a1008181-p5r5g\" (UID: \"b3eb393f-68f4-408c-a739-538d41b5b86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"
Oct 31 14:17:17.364718 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.364669    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3eb393f-68f4-408c-a739-538d41b5b86c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1008181-p5r5g\" (UID: \"b3eb393f-68f4-408c-a739-538d41b5b86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"
Oct 31 14:17:17.364884 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.364726    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcvxx\" (UniqueName: \"kubernetes.io/projected/b3eb393f-68f4-408c-a739-538d41b5b86c-kube-api-access-qcvxx\") pod \"maas-api-revoke-keys-e2e-ait-a1008181-p5r5g\" (UID: \"b3eb393f-68f4-408c-a739-538d41b5b86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"
Oct 31 14:17:17.364884 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.364810    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3eb393f-68f4-408c-a739-538d41b5b86c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1008181-p5r5g\" (UID: \"b3eb393f-68f4-408c-a739-538d41b5b86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"
Oct 31 14:17:17.365396 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.365370    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3eb393f-68f4-408c-a739-538d41b5b86c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1008181-p5r5g\" (UID: \"b3eb393f-68f4-408c-a739-538d41b5b86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"
Oct 31 14:17:17.367451 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.367425    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3eb393f-68f4-408c-a739-538d41b5b86c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1008181-p5r5g\" (UID: \"b3eb393f-68f4-408c-a739-538d41b5b86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"
Oct 31 14:17:17.371512 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:17:17.371492    4660 projected.go:196] Error preparing data for projected volume kube-api-access-qcvxx for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 31 14:17:17.371598 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:17:17.371551    4660 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3eb393f-68f4-408c-a739-538d41b5b86c-kube-api-access-qcvxx podName:b3eb393f-68f4-408c-a739-538d41b5b86c nodeName:}" failed. No retries permitted until 2026-10-31 14:17:17.871536168 +0000 UTC m=+4437.490195311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qcvxx" (UniqueName: "kubernetes.io/projected/b3eb393f-68f4-408c-a739-538d41b5b86c-kube-api-access-qcvxx") pod "maas-api-revoke-keys-e2e-ait-a1008181-p5r5g" (UID: "b3eb393f-68f4-408c-a739-538d41b5b86c") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 31 14:17:17.969839 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.969796    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcvxx\" (UniqueName: \"kubernetes.io/projected/b3eb393f-68f4-408c-a739-538d41b5b86c-kube-api-access-qcvxx\") pod \"maas-api-revoke-keys-e2e-ait-a1008181-p5r5g\" (UID: \"b3eb393f-68f4-408c-a739-538d41b5b86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"
Oct 31 14:17:17.983127 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:17.983095    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcvxx\" (UniqueName: \"kubernetes.io/projected/b3eb393f-68f4-408c-a739-538d41b5b86c-kube-api-access-qcvxx\") pod \"maas-api-revoke-keys-e2e-ait-a1008181-p5r5g\" (UID: \"b3eb393f-68f4-408c-a739-538d41b5b86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"
Oct 31 14:17:18.105428 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:18.105387    4660 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g" secret="" err="secret \"maas-api-cleanup-dockercfg-hxdwv\" not found"
Oct 31 14:17:18.105619 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:18.105447    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"
Oct 31 14:17:18.384444 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:18.384415    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"]
Oct 31 14:17:18.386655 ip-10-0-137-77 kubenswrapper[4660]: W1031 14:17:18.386631    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3eb393f_68f4_408c_a739_538d41b5b86c.slice/crio-75912578d665fc14d3d0a42a6ab1e5ee1f1cfc14e444657d521c205374452e8f WatchSource:0}: Error finding container 75912578d665fc14d3d0a42a6ab1e5ee1f1cfc14e444657d521c205374452e8f: Status 404 returned error can't find the container with id 75912578d665fc14d3d0a42a6ab1e5ee1f1cfc14e444657d521c205374452e8f
Oct 31 14:17:18.567894 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:18.567864    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g" event={"ID":"b3eb393f-68f4-408c-a739-538d41b5b86c","Type":"ContainerStarted","Data":"75912578d665fc14d3d0a42a6ab1e5ee1f1cfc14e444657d521c205374452e8f"}
Oct 31 14:17:19.575440 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:19.575347    4660 generic.go:356] "Generic (PLEG): container finished" podID="b3eb393f-68f4-408c-a739-538d41b5b86c" containerID="b68712caa09c770f27039c13491350a53993b08a5c02c16cbd43928f58a480bb" exitCode=0
Oct 31 14:17:19.575856 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:19.575439    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g" event={"ID":"b3eb393f-68f4-408c-a739-538d41b5b86c","Type":"ContainerDied","Data":"b68712caa09c770f27039c13491350a53993b08a5c02c16cbd43928f58a480bb"}
Oct 31 14:17:20.733088 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:17:20.733041    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:17:20.734586 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:20.734562    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"
Oct 31 14:17:20.795582 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:20.795544    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3eb393f-68f4-408c-a739-538d41b5b86c-kube-api-access-qcvxx\" (UniqueName: \"kubernetes.io/projected/b3eb393f-68f4-408c-a739-538d41b5b86c-kube-api-access-qcvxx\") pod \"b3eb393f-68f4-408c-a739-538d41b5b86c\" (UID: \"b3eb393f-68f4-408c-a739-538d41b5b86c\") "
Oct 31 14:17:20.795762 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:20.795598    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3eb393f-68f4-408c-a739-538d41b5b86c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3eb393f-68f4-408c-a739-538d41b5b86c-maas-api-openshift-ca\") pod \"b3eb393f-68f4-408c-a739-538d41b5b86c\" (UID: \"b3eb393f-68f4-408c-a739-538d41b5b86c\") "
Oct 31 14:17:20.795762 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:20.795663    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3eb393f-68f4-408c-a739-538d41b5b86c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3eb393f-68f4-408c-a739-538d41b5b86c-maas-api-xks-ca\") pod \"b3eb393f-68f4-408c-a739-538d41b5b86c\" (UID: \"b3eb393f-68f4-408c-a739-538d41b5b86c\") "
Oct 31 14:17:20.796005 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:20.795977    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3eb393f-68f4-408c-a739-538d41b5b86c-maas-api-openshift-ca" pod "b3eb393f-68f4-408c-a739-538d41b5b86c" (UID: "b3eb393f-68f4-408c-a739-538d41b5b86c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:17:20.797742 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:20.797713    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3eb393f-68f4-408c-a739-538d41b5b86c-maas-api-xks-ca" pod "b3eb393f-68f4-408c-a739-538d41b5b86c" (UID: "b3eb393f-68f4-408c-a739-538d41b5b86c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:17:20.797867 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:20.797806    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3eb393f-68f4-408c-a739-538d41b5b86c-kube-api-access-qcvxx" pod "b3eb393f-68f4-408c-a739-538d41b5b86c" (UID: "b3eb393f-68f4-408c-a739-538d41b5b86c"). InnerVolumeSpecName "kube-api-access-qcvxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:17:20.896948 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:20.896915    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3eb393f-68f4-408c-a739-538d41b5b86c-maas-api-xks-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:20.896948 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:20.896944    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcvxx\" (UniqueName: \"kubernetes.io/projected/b3eb393f-68f4-408c-a739-538d41b5b86c-kube-api-access-qcvxx\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:20.896948 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:20.896954    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3eb393f-68f4-408c-a739-538d41b5b86c-maas-api-openshift-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:21.587691 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:21.587656    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"
Oct 31 14:17:21.587879 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:21.587690    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g" event={"ID":"b3eb393f-68f4-408c-a739-538d41b5b86c","Type":"ContainerDied","Data":"75912578d665fc14d3d0a42a6ab1e5ee1f1cfc14e444657d521c205374452e8f"}
Oct 31 14:17:21.587879 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:21.587741    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75912578d665fc14d3d0a42a6ab1e5ee1f1cfc14e444657d521c205374452e8f"
Oct 31 14:17:28.210532 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:28.210491    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"]
Oct 31 14:17:28.214644 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:28.214613    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1008181-p5r5g"]
Oct 31 14:17:28.526203 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:17:28.526125    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3eb393f-68f4-408c-a739-538d41b5b86c" path="/var/lib/kubelet/pods/b3eb393f-68f4-408c-a739-538d41b5b86c/volumes"
Oct 31 14:17:50.732017 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:17:50.731976    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:18:20.733625 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:18:20.733583    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:18:21.515998 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:18:21.515968    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 14:18:21.519972 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:18:21.519948    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zwt49_2e19280d-a86c-48cd-b4e1-7cbe8308cb7d/extract/0.log"
Oct 31 14:18:21.522031 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:18:21.522007    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 14:18:21.525788 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:18:21.525769    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847az42gj_b84c453d-748d-4287-805c-09aa3dbcced8/extract/0.log"
Oct 31 14:18:21.530199 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:18:21.530181    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 14:18:21.534557 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:18:21.534539    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 14:18:50.732561 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:18:50.732517    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:19:14.284256 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.284176    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"]
Oct 31 14:19:14.284671 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.284601    4660 cpu_manager.go:414] "containerMap: removing container" podUID="b3eb393f-68f4-408c-a739-538d41b5b86c" containerName="revoke-keys"
Oct 31 14:19:14.284671 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.284615    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3eb393f-68f4-408c-a739-538d41b5b86c" containerName="revoke-keys"
Oct 31 14:19:14.284747 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.284708    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3eb393f-68f4-408c-a739-538d41b5b86c" containerName="revoke-keys"
Oct 31 14:19:14.287846 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.287828    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"
Oct 31 14:19:14.292259 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.292237    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:19:14.293382 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.293349    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 14:19:14.293382 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.293360    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 14:19:14.293382 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.293361    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrzbv\""
Oct 31 14:19:14.299026 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.298999    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"]
Oct 31 14:19:14.375688 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.375650    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qrhx\" (UniqueName: \"kubernetes.io/projected/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-kube-api-access-2qrhx\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk\" (UID: \"63e99d8a-c703-44ef-a7e1-34630a4ad4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"
Oct 31 14:19:14.375901 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.375815    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk\" (UID: \"63e99d8a-c703-44ef-a7e1-34630a4ad4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"
Oct 31 14:19:14.375901 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.375892    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk\" (UID: \"63e99d8a-c703-44ef-a7e1-34630a4ad4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"
Oct 31 14:19:14.477197 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.477150    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk\" (UID: \"63e99d8a-c703-44ef-a7e1-34630a4ad4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"
Oct 31 14:19:14.477424 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.477214    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk\" (UID: \"63e99d8a-c703-44ef-a7e1-34630a4ad4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"
Oct 31 14:19:14.477424 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.477269    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qrhx\" (UniqueName: \"kubernetes.io/projected/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-kube-api-access-2qrhx\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk\" (UID: \"63e99d8a-c703-44ef-a7e1-34630a4ad4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"
Oct 31 14:19:14.477986 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.477965    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk\" (UID: \"63e99d8a-c703-44ef-a7e1-34630a4ad4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"
Oct 31 14:19:14.479659 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.479638    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk\" (UID: \"63e99d8a-c703-44ef-a7e1-34630a4ad4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"
Oct 31 14:19:14.490572 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.490547    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qrhx\" (UniqueName: \"kubernetes.io/projected/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-kube-api-access-2qrhx\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk\" (UID: \"63e99d8a-c703-44ef-a7e1-34630a4ad4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"
Oct 31 14:19:14.602989 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.602909    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"
Oct 31 14:19:14.881437 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:14.881412    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"]
Oct 31 14:19:14.883654 ip-10-0-137-77 kubenswrapper[4660]: W1031 14:19:14.883626    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e99d8a_c703_44ef_a7e1_34630a4ad4a6.slice/crio-164755d2523b9671480009ec2e67fee6867ffad8b76358cb58a0617b511d2cda WatchSource:0}: Error finding container 164755d2523b9671480009ec2e67fee6867ffad8b76358cb58a0617b511d2cda: Status 404 returned error can't find the container with id 164755d2523b9671480009ec2e67fee6867ffad8b76358cb58a0617b511d2cda
Oct 31 14:19:15.304141 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:15.304101    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk" event={"ID":"63e99d8a-c703-44ef-a7e1-34630a4ad4a6","Type":"ContainerStarted","Data":"68020030218f423f3fd41d5e38ccb0490ba602f8cc415218f7b69804d114636e"}
Oct 31 14:19:15.304612 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:15.304199    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk" event={"ID":"63e99d8a-c703-44ef-a7e1-34630a4ad4a6","Type":"ContainerStarted","Data":"164755d2523b9671480009ec2e67fee6867ffad8b76358cb58a0617b511d2cda"}
Oct 31 14:19:15.325496 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:15.325448    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk" podStartSLOduration=1.325431858 podStartE2EDuration="1.325431858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:19:15.322718026 +0000 UTC m=+4554.941377191" watchObservedRunningTime="2026-10-31 14:19:15.325431858 +0000 UTC m=+4554.944091023"
Oct 31 14:19:16.313002 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:16.312966    4660 generic.go:356] "Generic (PLEG): container finished" podID="63e99d8a-c703-44ef-a7e1-34630a4ad4a6" containerID="68020030218f423f3fd41d5e38ccb0490ba602f8cc415218f7b69804d114636e" exitCode=0
Oct 31 14:19:16.313485 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:16.313039    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk" event={"ID":"63e99d8a-c703-44ef-a7e1-34630a4ad4a6","Type":"ContainerDied","Data":"68020030218f423f3fd41d5e38ccb0490ba602f8cc415218f7b69804d114636e"}
Oct 31 14:19:17.459718 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:17.459688    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"
Oct 31 14:19:17.607838 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:17.607748    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-kube-api-access-2qrhx\" (UniqueName: \"kubernetes.io/projected/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-kube-api-access-2qrhx\") pod \"63e99d8a-c703-44ef-a7e1-34630a4ad4a6\" (UID: \"63e99d8a-c703-44ef-a7e1-34630a4ad4a6\") "
Oct 31 14:19:17.607838 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:17.607824    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-maas-api-openshift-ca\") pod \"63e99d8a-c703-44ef-a7e1-34630a4ad4a6\" (UID: \"63e99d8a-c703-44ef-a7e1-34630a4ad4a6\") "
Oct 31 14:19:17.608039 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:17.607881    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-maas-api-xks-ca\") pod \"63e99d8a-c703-44ef-a7e1-34630a4ad4a6\" (UID: \"63e99d8a-c703-44ef-a7e1-34630a4ad4a6\") "
Oct 31 14:19:17.608257 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:17.608222    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-maas-api-openshift-ca" pod "63e99d8a-c703-44ef-a7e1-34630a4ad4a6" (UID: "63e99d8a-c703-44ef-a7e1-34630a4ad4a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:19:17.608451 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:17.608424    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-maas-api-openshift-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:17.610065 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:17.610040    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-kube-api-access-2qrhx" pod "63e99d8a-c703-44ef-a7e1-34630a4ad4a6" (UID: "63e99d8a-c703-44ef-a7e1-34630a4ad4a6"). InnerVolumeSpecName "kube-api-access-2qrhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:19:17.610158 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:17.610087    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-maas-api-xks-ca" pod "63e99d8a-c703-44ef-a7e1-34630a4ad4a6" (UID: "63e99d8a-c703-44ef-a7e1-34630a4ad4a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:19:17.709433 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:17.709395    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qrhx\" (UniqueName: \"kubernetes.io/projected/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-kube-api-access-2qrhx\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:17.709433 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:17.709428    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e99d8a-c703-44ef-a7e1-34630a4ad4a6-maas-api-xks-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:18.326195 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:18.326164    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"
Oct 31 14:19:18.326415 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:18.326205    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk" event={"ID":"63e99d8a-c703-44ef-a7e1-34630a4ad4a6","Type":"ContainerDied","Data":"164755d2523b9671480009ec2e67fee6867ffad8b76358cb58a0617b511d2cda"}
Oct 31 14:19:18.326415 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:18.326253    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="164755d2523b9671480009ec2e67fee6867ffad8b76358cb58a0617b511d2cda"
Oct 31 14:19:20.731146 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:19:20.731106    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:19:30.811103 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:30.811063    4660 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"]
Oct 31 14:19:30.814482 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:30.814453    4660 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c67e06-fk9fk"]
Oct 31 14:19:32.525911 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:32.525876    4660 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63e99d8a-c703-44ef-a7e1-34630a4ad4a6" path="/var/lib/kubelet/pods/63e99d8a-c703-44ef-a7e1-34630a4ad4a6/volumes"
Oct 31 14:19:48.119319 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:48.114411    4660 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 14:19:48.119319 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:48.114484    4660 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-2fpt9" podUID="ab9e59d3-2ac8-4381-98fa-22d3f25d682b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 14:19:49.683039 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.683006    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2"]
Oct 31 14:19:49.683455 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.683411    4660 cpu_manager.go:414] "containerMap: removing container" podUID="63e99d8a-c703-44ef-a7e1-34630a4ad4a6" containerName="revoke-keys"
Oct 31 14:19:49.683455 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.683424    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e99d8a-c703-44ef-a7e1-34630a4ad4a6" containerName="revoke-keys"
Oct 31 14:19:49.683534 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.683503    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e99d8a-c703-44ef-a7e1-34630a4ad4a6" containerName="revoke-keys"
Oct 31 14:19:49.687748 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.687729    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2"
Oct 31 14:19:49.694327 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.694272    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:19:49.694471 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.694420    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 14:19:49.694471 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.694451    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrzbv\""
Oct 31 14:19:49.694627 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.694480    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 14:19:49.696572 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.696550    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2"]
Oct 31 14:19:49.785416 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.785380    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjmdc\" (UniqueName: \"kubernetes.io/projected/e4c4e992-e0d1-479b-a01e-89847d28e760-kube-api-access-sjmdc\") pod \"maas-api-revoke-sub-f03629db-ncbm2\" (UID: \"e4c4e992-e0d1-479b-a01e-89847d28e760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2"
Oct 31 14:19:49.785593 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.785496    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c4e992-e0d1-479b-a01e-89847d28e760-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-ncbm2\" (UID: \"e4c4e992-e0d1-479b-a01e-89847d28e760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2"
Oct 31 14:19:49.785593 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.785579    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c4e992-e0d1-479b-a01e-89847d28e760-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-ncbm2\" (UID: \"e4c4e992-e0d1-479b-a01e-89847d28e760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2"
Oct 31 14:19:49.886421 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.886389    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c4e992-e0d1-479b-a01e-89847d28e760-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-ncbm2\" (UID: \"e4c4e992-e0d1-479b-a01e-89847d28e760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2"
Oct 31 14:19:49.886581 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.886446    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjmdc\" (UniqueName: \"kubernetes.io/projected/e4c4e992-e0d1-479b-a01e-89847d28e760-kube-api-access-sjmdc\") pod \"maas-api-revoke-sub-f03629db-ncbm2\" (UID: \"e4c4e992-e0d1-479b-a01e-89847d28e760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2"
Oct 31 14:19:49.886581 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.886506    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c4e992-e0d1-479b-a01e-89847d28e760-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-ncbm2\" (UID: \"e4c4e992-e0d1-479b-a01e-89847d28e760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2"
Oct 31 14:19:49.887041 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.887016    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c4e992-e0d1-479b-a01e-89847d28e760-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-ncbm2\" (UID: \"e4c4e992-e0d1-479b-a01e-89847d28e760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2"
Oct 31 14:19:49.888972 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.888945    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c4e992-e0d1-479b-a01e-89847d28e760-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-ncbm2\" (UID: \"e4c4e992-e0d1-479b-a01e-89847d28e760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2"
Oct 31 14:19:49.899954 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:49.899930    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjmdc\" (UniqueName: \"kubernetes.io/projected/e4c4e992-e0d1-479b-a01e-89847d28e760-kube-api-access-sjmdc\") pod \"maas-api-revoke-sub-f03629db-ncbm2\" (UID: \"e4c4e992-e0d1-479b-a01e-89847d28e760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2"
Oct 31 14:19:50.002242 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:50.002172    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2"
Oct 31 14:19:50.278646 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:50.278621    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2"]
Oct 31 14:19:50.280210 ip-10-0-137-77 kubenswrapper[4660]: W1031 14:19:50.280183    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c4e992_e0d1_479b_a01e_89847d28e760.slice/crio-9b8951b8a35e1c2cd1b4d40127e4892c8077a80bbf84de8a38cdb9d30d2a2408 WatchSource:0}: Error finding container 9b8951b8a35e1c2cd1b4d40127e4892c8077a80bbf84de8a38cdb9d30d2a2408: Status 404 returned error can't find the container with id 9b8951b8a35e1c2cd1b4d40127e4892c8077a80bbf84de8a38cdb9d30d2a2408
Oct 31 14:19:50.525067 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:50.525037    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2" event={"ID":"e4c4e992-e0d1-479b-a01e-89847d28e760","Type":"ContainerStarted","Data":"9b8951b8a35e1c2cd1b4d40127e4892c8077a80bbf84de8a38cdb9d30d2a2408"}
Oct 31 14:19:50.730799 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:19:50.730762    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:19:51.527568 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:51.527487    4660 generic.go:356] "Generic (PLEG): container finished" podID="e4c4e992-e0d1-479b-a01e-89847d28e760" containerID="5390fdec1a4b600ccc2576e7b48bce29dc46e24d2d7772da1a39ca24ed461e89" exitCode=0
Oct 31 14:19:51.527802 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:51.527567    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2" event={"ID":"e4c4e992-e0d1-479b-a01e-89847d28e760","Type":"ContainerDied","Data":"5390fdec1a4b600ccc2576e7b48bce29dc46e24d2d7772da1a39ca24ed461e89"}
Oct 31 14:19:52.670405 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:52.670373    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2"
Oct 31 14:19:52.712712 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:52.712684    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4c4e992-e0d1-479b-a01e-89847d28e760-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c4e992-e0d1-479b-a01e-89847d28e760-maas-api-xks-ca\") pod \"e4c4e992-e0d1-479b-a01e-89847d28e760\" (UID: \"e4c4e992-e0d1-479b-a01e-89847d28e760\") "
Oct 31 14:19:52.712851 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:52.712733    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c4e992-e0d1-479b-a01e-89847d28e760-kube-api-access-sjmdc\" (UniqueName: \"kubernetes.io/projected/e4c4e992-e0d1-479b-a01e-89847d28e760-kube-api-access-sjmdc\") pod \"e4c4e992-e0d1-479b-a01e-89847d28e760\" (UID: \"e4c4e992-e0d1-479b-a01e-89847d28e760\") "
Oct 31 14:19:52.712851 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:52.712814    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c4e992-e0d1-479b-a01e-89847d28e760-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c4e992-e0d1-479b-a01e-89847d28e760-maas-api-openshift-ca\") pod \"e4c4e992-e0d1-479b-a01e-89847d28e760\" (UID: \"e4c4e992-e0d1-479b-a01e-89847d28e760\") "
Oct 31 14:19:52.713216 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:52.713187    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c4e992-e0d1-479b-a01e-89847d28e760-maas-api-openshift-ca" pod "e4c4e992-e0d1-479b-a01e-89847d28e760" (UID: "e4c4e992-e0d1-479b-a01e-89847d28e760"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:19:52.714914 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:52.714888    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c4e992-e0d1-479b-a01e-89847d28e760-maas-api-xks-ca" pod "e4c4e992-e0d1-479b-a01e-89847d28e760" (UID: "e4c4e992-e0d1-479b-a01e-89847d28e760"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:19:52.715115 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:52.715086    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c4e992-e0d1-479b-a01e-89847d28e760-kube-api-access-sjmdc" pod "e4c4e992-e0d1-479b-a01e-89847d28e760" (UID: "e4c4e992-e0d1-479b-a01e-89847d28e760"). InnerVolumeSpecName "kube-api-access-sjmdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:19:52.814419 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:52.814344    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c4e992-e0d1-479b-a01e-89847d28e760-maas-api-xks-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:52.814419 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:52.814373    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjmdc\" (UniqueName: \"kubernetes.io/projected/e4c4e992-e0d1-479b-a01e-89847d28e760-kube-api-access-sjmdc\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:52.814419 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:52.814385    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c4e992-e0d1-479b-a01e-89847d28e760-maas-api-openshift-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:53.539329 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:53.539284    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2"
Oct 31 14:19:53.539329 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:53.539321    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ncbm2" event={"ID":"e4c4e992-e0d1-479b-a01e-89847d28e760","Type":"ContainerDied","Data":"9b8951b8a35e1c2cd1b4d40127e4892c8077a80bbf84de8a38cdb9d30d2a2408"}
Oct 31 14:19:53.539557 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:19:53.539364    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b8951b8a35e1c2cd1b4d40127e4892c8077a80bbf84de8a38cdb9d30d2a2408"
Oct 31 14:20:05.660356 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.660322    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn"]
Oct 31 14:20:05.660776 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.660758    4660 cpu_manager.go:414] "containerMap: removing container" podUID="e4c4e992-e0d1-479b-a01e-89847d28e760" containerName="revoke-keys"
Oct 31 14:20:05.660823 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.660778    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c4e992-e0d1-479b-a01e-89847d28e760" containerName="revoke-keys"
Oct 31 14:20:05.660870 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.660860    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c4e992-e0d1-479b-a01e-89847d28e760" containerName="revoke-keys"
Oct 31 14:20:05.663504 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.663488    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn"
Oct 31 14:20:05.671082 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.671054    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:20:05.671212 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.671085    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrzbv\""
Oct 31 14:20:05.673846 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.673818    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn"]
Oct 31 14:20:05.682550 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.682528    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 14:20:05.682667 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.682577    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 14:20:05.687993 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.687970    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr527\" (UniqueName: \"kubernetes.io/projected/e3db437b-e11e-45c5-83dd-beb49a13500f-kube-api-access-gr527\") pod \"maas-api-revoke-sub-0165d7d2-2rvzn\" (UID: \"e3db437b-e11e-45c5-83dd-beb49a13500f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn"
Oct 31 14:20:05.688100 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.688004    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3db437b-e11e-45c5-83dd-beb49a13500f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2rvzn\" (UID: \"e3db437b-e11e-45c5-83dd-beb49a13500f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn"
Oct 31 14:20:05.688143 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.688117    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3db437b-e11e-45c5-83dd-beb49a13500f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2rvzn\" (UID: \"e3db437b-e11e-45c5-83dd-beb49a13500f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn"
Oct 31 14:20:05.788939 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.788907    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr527\" (UniqueName: \"kubernetes.io/projected/e3db437b-e11e-45c5-83dd-beb49a13500f-kube-api-access-gr527\") pod \"maas-api-revoke-sub-0165d7d2-2rvzn\" (UID: \"e3db437b-e11e-45c5-83dd-beb49a13500f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn"
Oct 31 14:20:05.789103 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.788953    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3db437b-e11e-45c5-83dd-beb49a13500f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2rvzn\" (UID: \"e3db437b-e11e-45c5-83dd-beb49a13500f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn"
Oct 31 14:20:05.789103 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.789055    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3db437b-e11e-45c5-83dd-beb49a13500f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2rvzn\" (UID: \"e3db437b-e11e-45c5-83dd-beb49a13500f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn"
Oct 31 14:20:05.789651 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.789629    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3db437b-e11e-45c5-83dd-beb49a13500f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2rvzn\" (UID: \"e3db437b-e11e-45c5-83dd-beb49a13500f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn"
Oct 31 14:20:05.791469 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.791448    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3db437b-e11e-45c5-83dd-beb49a13500f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2rvzn\" (UID: \"e3db437b-e11e-45c5-83dd-beb49a13500f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn"
Oct 31 14:20:05.801894 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.801870    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr527\" (UniqueName: \"kubernetes.io/projected/e3db437b-e11e-45c5-83dd-beb49a13500f-kube-api-access-gr527\") pod \"maas-api-revoke-sub-0165d7d2-2rvzn\" (UID: \"e3db437b-e11e-45c5-83dd-beb49a13500f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn"
Oct 31 14:20:05.979946 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:05.979877    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn"
Oct 31 14:20:06.261031 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:06.261003    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn"]
Oct 31 14:20:06.263491 ip-10-0-137-77 kubenswrapper[4660]: W1031 14:20:06.263458    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3db437b_e11e_45c5_83dd_beb49a13500f.slice/crio-2ad09b45a8173d184bd18960b0acf22cc9a889b8977c2c20f53eb038719e174f WatchSource:0}: Error finding container 2ad09b45a8173d184bd18960b0acf22cc9a889b8977c2c20f53eb038719e174f: Status 404 returned error can't find the container with id 2ad09b45a8173d184bd18960b0acf22cc9a889b8977c2c20f53eb038719e174f
Oct 31 14:20:06.610248 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:06.610166    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn" event={"ID":"e3db437b-e11e-45c5-83dd-beb49a13500f","Type":"ContainerStarted","Data":"1105d40ec1bb9f6cda3a29907d68028cf1bfed1e4d9a8c2fcf7b176b83fdaeb2"}
Oct 31 14:20:06.610248 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:06.610218    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn" event={"ID":"e3db437b-e11e-45c5-83dd-beb49a13500f","Type":"ContainerStarted","Data":"2ad09b45a8173d184bd18960b0acf22cc9a889b8977c2c20f53eb038719e174f"}
Oct 31 14:20:07.617922 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:07.617883    4660 generic.go:356] "Generic (PLEG): container finished" podID="e3db437b-e11e-45c5-83dd-beb49a13500f" containerID="1105d40ec1bb9f6cda3a29907d68028cf1bfed1e4d9a8c2fcf7b176b83fdaeb2" exitCode=0
Oct 31 14:20:07.618346 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:07.617935    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn" event={"ID":"e3db437b-e11e-45c5-83dd-beb49a13500f","Type":"ContainerDied","Data":"1105d40ec1bb9f6cda3a29907d68028cf1bfed1e4d9a8c2fcf7b176b83fdaeb2"}
Oct 31 14:20:08.761999 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:08.761971    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn"
Oct 31 14:20:08.821610 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:08.821580    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3db437b-e11e-45c5-83dd-beb49a13500f-kube-api-access-gr527\" (UniqueName: \"kubernetes.io/projected/e3db437b-e11e-45c5-83dd-beb49a13500f-kube-api-access-gr527\") pod \"e3db437b-e11e-45c5-83dd-beb49a13500f\" (UID: \"e3db437b-e11e-45c5-83dd-beb49a13500f\") "
Oct 31 14:20:08.821802 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:08.821643    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3db437b-e11e-45c5-83dd-beb49a13500f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3db437b-e11e-45c5-83dd-beb49a13500f-maas-api-xks-ca\") pod \"e3db437b-e11e-45c5-83dd-beb49a13500f\" (UID: \"e3db437b-e11e-45c5-83dd-beb49a13500f\") "
Oct 31 14:20:08.821802 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:08.821711    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3db437b-e11e-45c5-83dd-beb49a13500f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3db437b-e11e-45c5-83dd-beb49a13500f-maas-api-openshift-ca\") pod \"e3db437b-e11e-45c5-83dd-beb49a13500f\" (UID: \"e3db437b-e11e-45c5-83dd-beb49a13500f\") "
Oct 31 14:20:08.822060 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:08.822036    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3db437b-e11e-45c5-83dd-beb49a13500f-maas-api-openshift-ca" pod "e3db437b-e11e-45c5-83dd-beb49a13500f" (UID: "e3db437b-e11e-45c5-83dd-beb49a13500f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:20:08.823719 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:08.823690    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3db437b-e11e-45c5-83dd-beb49a13500f-kube-api-access-gr527" pod "e3db437b-e11e-45c5-83dd-beb49a13500f" (UID: "e3db437b-e11e-45c5-83dd-beb49a13500f"). InnerVolumeSpecName "kube-api-access-gr527". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:20:08.823719 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:08.823705    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3db437b-e11e-45c5-83dd-beb49a13500f-maas-api-xks-ca" pod "e3db437b-e11e-45c5-83dd-beb49a13500f" (UID: "e3db437b-e11e-45c5-83dd-beb49a13500f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:20:08.923127 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:08.923053    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3db437b-e11e-45c5-83dd-beb49a13500f-maas-api-xks-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:20:08.923127 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:08.923083    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3db437b-e11e-45c5-83dd-beb49a13500f-maas-api-openshift-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:20:08.923127 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:08.923094    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr527\" (UniqueName: \"kubernetes.io/projected/e3db437b-e11e-45c5-83dd-beb49a13500f-kube-api-access-gr527\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:20:09.631136 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:09.631107    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn"
Oct 31 14:20:09.631325 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:09.631107    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2rvzn" event={"ID":"e3db437b-e11e-45c5-83dd-beb49a13500f","Type":"ContainerDied","Data":"2ad09b45a8173d184bd18960b0acf22cc9a889b8977c2c20f53eb038719e174f"}
Oct 31 14:20:09.631325 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:09.631236    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ad09b45a8173d184bd18960b0acf22cc9a889b8977c2c20f53eb038719e174f"
Oct 31 14:20:16.497387 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.497348    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd"]
Oct 31 14:20:16.497816 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.497800    4660 cpu_manager.go:414] "containerMap: removing container" podUID="e3db437b-e11e-45c5-83dd-beb49a13500f" containerName="revoke-keys"
Oct 31 14:20:16.497865 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.497816    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3db437b-e11e-45c5-83dd-beb49a13500f" containerName="revoke-keys"
Oct 31 14:20:16.497902 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.497883    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3db437b-e11e-45c5-83dd-beb49a13500f" containerName="revoke-keys"
Oct 31 14:20:16.501179 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.501161    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd"
Oct 31 14:20:16.509871 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.509368    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 14:20:16.509871 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.509430    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:20:16.509871 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.509519    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrzbv\""
Oct 31 14:20:16.509871 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.509574    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 14:20:16.510940 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.510913    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd"]
Oct 31 14:20:16.591973 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.591942    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-kshvd\" (UID: \"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd"
Oct 31 14:20:16.592161 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.592009    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t6q4\" (UniqueName: \"kubernetes.io/projected/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-kube-api-access-9t6q4\") pod \"maas-api-revoke-sub-6b19338f-kshvd\" (UID: \"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd"
Oct 31 14:20:16.592263 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.592242    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-kshvd\" (UID: \"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd"
Oct 31 14:20:16.693332 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.693275    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-kshvd\" (UID: \"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd"
Oct 31 14:20:16.693515 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.693380    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-kshvd\" (UID: \"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd"
Oct 31 14:20:16.693515 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.693421    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t6q4\" (UniqueName: \"kubernetes.io/projected/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-kube-api-access-9t6q4\") pod \"maas-api-revoke-sub-6b19338f-kshvd\" (UID: \"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd"
Oct 31 14:20:16.694093 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.694070    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-kshvd\" (UID: \"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd"
Oct 31 14:20:16.695583 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.695562    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-kshvd\" (UID: \"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd"
Oct 31 14:20:16.706636 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.706610    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t6q4\" (UniqueName: \"kubernetes.io/projected/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-kube-api-access-9t6q4\") pod \"maas-api-revoke-sub-6b19338f-kshvd\" (UID: \"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd"
Oct 31 14:20:16.818473 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:16.818389    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd"
Oct 31 14:20:17.097420 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:17.097392    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd"]
Oct 31 14:20:17.099494 ip-10-0-137-77 kubenswrapper[4660]: W1031 14:20:17.099465    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cd75d7_3ffe_48ef_97cb_0e4a3caada9b.slice/crio-1a20c0f276bc82f62b57b7a87a3729d58518a98a8aa6d858691bef67f47b824c WatchSource:0}: Error finding container 1a20c0f276bc82f62b57b7a87a3729d58518a98a8aa6d858691bef67f47b824c: Status 404 returned error can't find the container with id 1a20c0f276bc82f62b57b7a87a3729d58518a98a8aa6d858691bef67f47b824c
Oct 31 14:20:17.679731 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:17.679695    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd" event={"ID":"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b","Type":"ContainerStarted","Data":"f8bad32ec13cd6117e8dfbc23c7fd75ed2c278e681c97c84726ff65f201b451a"}
Oct 31 14:20:17.680139 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:17.679748    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd" event={"ID":"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b","Type":"ContainerStarted","Data":"1a20c0f276bc82f62b57b7a87a3729d58518a98a8aa6d858691bef67f47b824c"}
Oct 31 14:20:17.700479 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:17.700427    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd" podStartSLOduration=1.700413938 podStartE2EDuration="1.700413938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:20:17.698694004 +0000 UTC m=+4617.317353169" watchObservedRunningTime="2026-10-31 14:20:17.700413938 +0000 UTC m=+4617.319073102"
Oct 31 14:20:18.687325 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:18.687274    4660 generic.go:356] "Generic (PLEG): container finished" podID="85cd75d7-3ffe-48ef-97cb-0e4a3caada9b" containerID="f8bad32ec13cd6117e8dfbc23c7fd75ed2c278e681c97c84726ff65f201b451a" exitCode=0
Oct 31 14:20:18.687744 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:18.687362    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd" event={"ID":"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b","Type":"ContainerDied","Data":"f8bad32ec13cd6117e8dfbc23c7fd75ed2c278e681c97c84726ff65f201b451a"}
Oct 31 14:20:19.824388 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:19.824364    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd"
Oct 31 14:20:19.922080 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:19.922049    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-maas-api-xks-ca\") pod \"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b\" (UID: \"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b\") "
Oct 31 14:20:19.922215 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:19.922116    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-maas-api-openshift-ca\") pod \"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b\" (UID: \"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b\") "
Oct 31 14:20:19.922273 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:19.922219    4660 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-kube-api-access-9t6q4\" (UniqueName: \"kubernetes.io/projected/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-kube-api-access-9t6q4\") pod \"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b\" (UID: \"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b\") "
Oct 31 14:20:19.922539 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:19.922505    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-maas-api-openshift-ca" pod "85cd75d7-3ffe-48ef-97cb-0e4a3caada9b" (UID: "85cd75d7-3ffe-48ef-97cb-0e4a3caada9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:20:19.924289 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:19.924266    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-maas-api-xks-ca" pod "85cd75d7-3ffe-48ef-97cb-0e4a3caada9b" (UID: "85cd75d7-3ffe-48ef-97cb-0e4a3caada9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:20:19.924403 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:19.924360    4660 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-kube-api-access-9t6q4" pod "85cd75d7-3ffe-48ef-97cb-0e4a3caada9b" (UID: "85cd75d7-3ffe-48ef-97cb-0e4a3caada9b"). InnerVolumeSpecName "kube-api-access-9t6q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:20:20.023186 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:20.023097    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-maas-api-openshift-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:20:20.023186 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:20.023136    4660 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t6q4\" (UniqueName: \"kubernetes.io/projected/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-kube-api-access-9t6q4\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:20:20.023186 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:20.023150    4660 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85cd75d7-3ffe-48ef-97cb-0e4a3caada9b-maas-api-xks-ca\") on node \"ip-10-0-137-77.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:20:20.699216 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:20.699183    4660 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd"
Oct 31 14:20:20.699399 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:20.699219    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kshvd" event={"ID":"85cd75d7-3ffe-48ef-97cb-0e4a3caada9b","Type":"ContainerDied","Data":"1a20c0f276bc82f62b57b7a87a3729d58518a98a8aa6d858691bef67f47b824c"}
Oct 31 14:20:20.699399 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:20.699268    4660 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a20c0f276bc82f62b57b7a87a3729d58518a98a8aa6d858691bef67f47b824c"
Oct 31 14:20:20.730618 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:20:20.730579    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:20:27.806148 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:27.806095    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f57567d7b-4vrrf_2d5c423b-4fd2-408b-acce-dfa021e19cb4/authorino/0.log"
Oct 31 14:20:38.983980 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:38.983872    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wr6rq_f5aeef9b-2b39-4e37-88ef-fadf328165d8/manager/0.log"
Oct 31 14:20:39.194285 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:39.194244    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-dk7st_1c508770-dcf5-4daf-b831-78f03a35e81b/manager/0.log"
Oct 31 14:20:41.329084 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:41.329053    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-2rvzn_e3db437b-e11e-45c5-83dd-beb49a13500f/revoke-keys/0.log"
Oct 31 14:20:42.029241 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:42.029205    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-kshvd_85cd75d7-3ffe-48ef-97cb-0e4a3caada9b/revoke-keys/0.log"
Oct 31 14:20:42.214241 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:42.214204    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-ncbm2_e4c4e992-e0d1-479b-a01e-89847d28e760/revoke-keys/0.log"
Oct 31 14:20:43.908730 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:43.908687    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f57567d7b-4vrrf_2d5c423b-4fd2-408b-acce-dfa021e19cb4/authorino/0.log"
Oct 31 14:20:44.149241 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:44.149205    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-grwkn_c831b6ce-162b-47af-a912-d53dfea59e2c/manager/0.log"
Oct 31 14:20:44.333252 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:44.333170    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9lrk5_227ac746-4ef3-4dde-aaaa-ba5f7c0e89c0/manager/0.log"
Oct 31 14:20:44.515984 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:44.515954    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-wl4g7_8cc01303-cfd7-44cc-a013-50501b81c268/kuadrant-console-plugin/0.log"
Oct 31 14:20:45.146487 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:45.146448    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-zvfgt_8b5c39b4-e8f2-45dc-8c45-3dd03ac475fa/manager/0.log"
Oct 31 14:20:45.927685 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:45.927653    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf_35162b05-708e-4e60-8fe0-7d78a8ca24ae/istio-proxy/0.log"
Oct 31 14:20:46.123944 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:46.123911    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-59fjz_97903974-f732-4729-9d22-d4d319663078/discovery/0.log"
Oct 31 14:20:46.648907 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:46.648877    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-2fpt9_ab9e59d3-2ac8-4381-98fa-22d3f25d682b/istio-proxy/1.log"
Oct 31 14:20:46.843930 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:46.843891    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-5z724_541446ef-8e36-4be7-9a16-b4005f6c250a/istio-proxy/0.log"
Oct 31 14:20:47.571969 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:47.571933    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-gwl2w_876ecf6e-3c48-4f96-8eaa-d7d5196be2ca/istio-proxy/0.log"
Oct 31 14:20:48.502792 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:48.502760    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6_26126910-4393-4b5e-b788-71db446afb82/istio-proxy/0.log"
Oct 31 14:20:49.032661 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:49.032628    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-848dcf9575-7trg4_c95a45a3-7757-4843-9161-6cdbf64385cd/router/0.log"
Oct 31 14:20:49.469624 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:49.469586    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-2fpt9_ab9e59d3-2ac8-4381-98fa-22d3f25d682b/istio-proxy/0.log"
Oct 31 14:20:50.732040 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:20:50.731998    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:20:52.195532 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:20:52.195497    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f57567d7b-4vrrf_2d5c423b-4fd2-408b-acce-dfa021e19cb4/authorino/0.log"
Oct 31 14:21:01.799100 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:01.799062    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wr6rq_f5aeef9b-2b39-4e37-88ef-fadf328165d8/manager/0.log"
Oct 31 14:21:02.015989 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:02.015951    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-dk7st_1c508770-dcf5-4daf-b831-78f03a35e81b/manager/0.log"
Oct 31 14:21:03.707080 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:03.707047    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-2rvzn_e3db437b-e11e-45c5-83dd-beb49a13500f/revoke-keys/0.log"
Oct 31 14:21:04.420104 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:04.420070    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-kshvd_85cd75d7-3ffe-48ef-97cb-0e4a3caada9b/revoke-keys/0.log"
Oct 31 14:21:04.589207 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:04.589171    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-ncbm2_e4c4e992-e0d1-479b-a01e-89847d28e760/revoke-keys/0.log"
Oct 31 14:21:05.879971 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:05.879934    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f57567d7b-4vrrf_2d5c423b-4fd2-408b-acce-dfa021e19cb4/authorino/0.log"
Oct 31 14:21:06.105885 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:06.105849    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-grwkn_c831b6ce-162b-47af-a912-d53dfea59e2c/manager/0.log"
Oct 31 14:21:06.294693 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:06.294615    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9lrk5_227ac746-4ef3-4dde-aaaa-ba5f7c0e89c0/manager/0.log"
Oct 31 14:21:06.475794 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:06.475764    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-wl4g7_8cc01303-cfd7-44cc-a013-50501b81c268/kuadrant-console-plugin/0.log"
Oct 31 14:21:07.054949 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:07.054914    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-zvfgt_8b5c39b4-e8f2-45dc-8c45-3dd03ac475fa/manager/0.log"
Oct 31 14:21:07.594650 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:07.594609    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf_35162b05-708e-4e60-8fe0-7d78a8ca24ae/istio-proxy/0.log"
Oct 31 14:21:07.785420 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:07.785386    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-59fjz_97903974-f732-4729-9d22-d4d319663078/discovery/0.log"
Oct 31 14:21:08.377493 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:08.377461    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-2fpt9_ab9e59d3-2ac8-4381-98fa-22d3f25d682b/istio-proxy/1.log"
Oct 31 14:21:08.589801 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:08.589749    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-5z724_541446ef-8e36-4be7-9a16-b4005f6c250a/istio-proxy/0.log"
Oct 31 14:21:09.341104 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:09.341065    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-gwl2w_876ecf6e-3c48-4f96-8eaa-d7d5196be2ca/istio-proxy/0.log"
Oct 31 14:21:10.310466 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:10.310420    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6_26126910-4393-4b5e-b788-71db446afb82/istio-proxy/0.log"
Oct 31 14:21:10.869413 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:21:10.869376    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-848dcf9575-7trg4_c95a45a3-7757-4843-9161-6cdbf64385cd/router/0.log"
Oct 31 14:21:20.732380 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:21:20.732342    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:21:50.731673 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:21:50.731558    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:22:20.731775 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:22:20.731684    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:22:41.492040 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:41.492006    4660 ???:1] "http: TLS handshake error from 10.0.137.77:51518: EOF"
Oct 31 14:22:41.501224 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:41.501192    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hrj9c_91204e03-5b1b-4084-a699-ac7b59db4be5/global-pull-secret-syncer/0.log"
Oct 31 14:22:41.550238 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:41.550209    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-dswrj_40395d51-6cc7-44b3-8cad-d4a44489e25f/connection-checker/0.log"
Oct 31 14:22:41.580961 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:41.580933    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-sqxqp_8c98a759-eb02-4d1b-a9f8-f1cb77e1590a/connection-checker/0.log"
Oct 31 14:22:41.610309 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:41.610268    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wvdhx_ad027e2c-c49c-4c9b-8253-12acc36a1c9c/connection-checker/0.log"
Oct 31 14:22:41.746533 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:41.746448    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mttz6_dc5cb602-ea83-47e3-8966-b832277cb06c/konnectivity-agent/0.log"
Oct 31 14:22:41.923804 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:41.923768    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-77.us-east-2.compute.internal_e08a66f57394b6ed04cdde42605f9314/haproxy/0.log"
Oct 31 14:22:49.797490 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:49.797424    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f57567d7b-4vrrf_2d5c423b-4fd2-408b-acce-dfa021e19cb4/authorino/0.log"
Oct 31 14:22:49.831552 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:49.831514    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-grwkn_c831b6ce-162b-47af-a912-d53dfea59e2c/manager/0.log"
Oct 31 14:22:49.862201 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:49.862170    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9lrk5_227ac746-4ef3-4dde-aaaa-ba5f7c0e89c0/manager/0.log"
Oct 31 14:22:49.901647 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:49.901616    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-wl4g7_8cc01303-cfd7-44cc-a013-50501b81c268/kuadrant-console-plugin/0.log"
Oct 31 14:22:50.204699 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:50.204661    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-zvfgt_8b5c39b4-e8f2-45dc-8c45-3dd03ac475fa/manager/0.log"
Oct 31 14:22:50.732276 ip-10-0-137-77 kubenswrapper[4660]: E1031 14:22:50.732240    4660 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9hjjp" containerName="extractor"
Oct 31 14:22:51.435380 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.435337    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-44698_bf8d1062-0bb5-46f2-98bf-e4d52689294c/cluster-monitoring-operator/0.log"
Oct 31 14:22:51.438182 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.438150    4660 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"]
Oct 31 14:22:51.438621 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.438606    4660 cpu_manager.go:414] "containerMap: removing container" podUID="85cd75d7-3ffe-48ef-97cb-0e4a3caada9b" containerName="revoke-keys"
Oct 31 14:22:51.438672 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.438622    4660 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85cd75d7-3ffe-48ef-97cb-0e4a3caada9b" containerName="revoke-keys"
Oct 31 14:22:51.438709 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.438701    4660 memory_manager.go:365] "RemoveStaleState removing state" podUID="85cd75d7-3ffe-48ef-97cb-0e4a3caada9b" containerName="revoke-keys"
Oct 31 14:22:51.441722 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.441705    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.450599 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.450574    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"]
Oct 31 14:22:51.529144 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.529106    4660 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bkxgk\"/\"default-dockercfg-smppd\""
Oct 31 14:22:51.529629 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.529611    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bkxgk\"/\"openshift-service-ca.crt\""
Oct 31 14:22:51.529701 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.529683    4660 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bkxgk\"/\"kube-root-ca.crt\""
Oct 31 14:22:51.583121 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.583084    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82bdc239-0515-4002-bb69-3a8a3fbf1767-proc\") pod \"perf-node-gather-daemonset-8kg4j\" (UID: \"82bdc239-0515-4002-bb69-3a8a3fbf1767\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.583278 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.583227    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c78x\" (UniqueName: \"kubernetes.io/projected/82bdc239-0515-4002-bb69-3a8a3fbf1767-kube-api-access-9c78x\") pod \"perf-node-gather-daemonset-8kg4j\" (UID: \"82bdc239-0515-4002-bb69-3a8a3fbf1767\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.583278 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.583269    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82bdc239-0515-4002-bb69-3a8a3fbf1767-podres\") pod \"perf-node-gather-daemonset-8kg4j\" (UID: \"82bdc239-0515-4002-bb69-3a8a3fbf1767\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.583410 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.583312    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82bdc239-0515-4002-bb69-3a8a3fbf1767-lib-modules\") pod \"perf-node-gather-daemonset-8kg4j\" (UID: \"82bdc239-0515-4002-bb69-3a8a3fbf1767\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.583410 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.583384    4660 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82bdc239-0515-4002-bb69-3a8a3fbf1767-sys\") pod \"perf-node-gather-daemonset-8kg4j\" (UID: \"82bdc239-0515-4002-bb69-3a8a3fbf1767\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.684338 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.684288    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82bdc239-0515-4002-bb69-3a8a3fbf1767-podres\") pod \"perf-node-gather-daemonset-8kg4j\" (UID: \"82bdc239-0515-4002-bb69-3a8a3fbf1767\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.684520 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.684359    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82bdc239-0515-4002-bb69-3a8a3fbf1767-lib-modules\") pod \"perf-node-gather-daemonset-8kg4j\" (UID: \"82bdc239-0515-4002-bb69-3a8a3fbf1767\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.684520 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.684475    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82bdc239-0515-4002-bb69-3a8a3fbf1767-sys\") pod \"perf-node-gather-daemonset-8kg4j\" (UID: \"82bdc239-0515-4002-bb69-3a8a3fbf1767\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.684598 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.684520    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82bdc239-0515-4002-bb69-3a8a3fbf1767-proc\") pod \"perf-node-gather-daemonset-8kg4j\" (UID: \"82bdc239-0515-4002-bb69-3a8a3fbf1767\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.684634 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.684621    4660 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c78x\" (UniqueName: \"kubernetes.io/projected/82bdc239-0515-4002-bb69-3a8a3fbf1767-kube-api-access-9c78x\") pod \"perf-node-gather-daemonset-8kg4j\" (UID: \"82bdc239-0515-4002-bb69-3a8a3fbf1767\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.685062 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.685043    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82bdc239-0515-4002-bb69-3a8a3fbf1767-proc\") pod \"perf-node-gather-daemonset-8kg4j\" (UID: \"82bdc239-0515-4002-bb69-3a8a3fbf1767\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.685108 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.685055    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82bdc239-0515-4002-bb69-3a8a3fbf1767-lib-modules\") pod \"perf-node-gather-daemonset-8kg4j\" (UID: \"82bdc239-0515-4002-bb69-3a8a3fbf1767\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.685108 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.685091    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82bdc239-0515-4002-bb69-3a8a3fbf1767-podres\") pod \"perf-node-gather-daemonset-8kg4j\" (UID: \"82bdc239-0515-4002-bb69-3a8a3fbf1767\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.685172 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.685091    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82bdc239-0515-4002-bb69-3a8a3fbf1767-sys\") pod \"perf-node-gather-daemonset-8kg4j\" (UID: \"82bdc239-0515-4002-bb69-3a8a3fbf1767\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.697582 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.697522    4660 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c78x\" (UniqueName: \"kubernetes.io/projected/82bdc239-0515-4002-bb69-3a8a3fbf1767-kube-api-access-9c78x\") pod \"perf-node-gather-daemonset-8kg4j\" (UID: \"82bdc239-0515-4002-bb69-3a8a3fbf1767\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.756055 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.756019    4660 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:51.801895 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.801864    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rgjfb_4e43df86-2b4a-4deb-ac8f-d39ba65c9f73/node-exporter/0.log"
Oct 31 14:22:51.832648 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.832555    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rgjfb_4e43df86-2b4a-4deb-ac8f-d39ba65c9f73/kube-rbac-proxy/0.log"
Oct 31 14:22:51.859967 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:51.859942    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rgjfb_4e43df86-2b4a-4deb-ac8f-d39ba65c9f73/init-textfile/0.log"
Oct 31 14:22:52.036501 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:52.036470    4660 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"]
Oct 31 14:22:52.038211 ip-10-0-137-77 kubenswrapper[4660]: W1031 14:22:52.038182    4660 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod82bdc239_0515_4002_bb69_3a8a3fbf1767.slice/crio-a9720caf4d9b9790a5348ec31c96635bc306e4d8debfa53f10eae7f2f9781e9a WatchSource:0}: Error finding container a9720caf4d9b9790a5348ec31c96635bc306e4d8debfa53f10eae7f2f9781e9a: Status 404 returned error can't find the container with id a9720caf4d9b9790a5348ec31c96635bc306e4d8debfa53f10eae7f2f9781e9a
Oct 31 14:22:52.630124 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:52.630083    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j" event={"ID":"82bdc239-0515-4002-bb69-3a8a3fbf1767","Type":"ContainerStarted","Data":"9923c00005a210a87cf8a1277e8a90ab10c85a09715c6d1d240232a971383e5f"}
Oct 31 14:22:52.630585 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:52.630140    4660 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j" event={"ID":"82bdc239-0515-4002-bb69-3a8a3fbf1767","Type":"ContainerStarted","Data":"a9720caf4d9b9790a5348ec31c96635bc306e4d8debfa53f10eae7f2f9781e9a"}
Oct 31 14:22:52.630585 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:52.630221    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:52.651261 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:52.651211    4660 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j" podStartSLOduration=1.651198955 podStartE2EDuration="1.651198955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:22:52.650200383 +0000 UTC m=+4772.268859549" watchObservedRunningTime="2026-10-31 14:22:52.651198955 +0000 UTC m=+4772.269858118"
Oct 31 14:22:53.817410 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:53.817376    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-swrc7_73811006-7011-4339-800d-5f1b8a9bbf88/networking-console-plugin/0.log"
Oct 31 14:22:54.411590 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:54.411555    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/2.log"
Oct 31 14:22:54.421033 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:54.421005    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7v9w9_4504b20c-a7d8-49e9-a572-9004fce4af1d/console-operator/3.log"
Oct 31 14:22:55.617484 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:55.617450    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-w7wlc_a096aa5d-63cc-41a6-ad78-bc64d2d33e94/volume-data-source-validator/0.log"
Oct 31 14:22:56.963306 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:56.963269    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pztwr_bdce0f75-852b-49b6-9da3-f662bb82bf94/dns/0.log"
Oct 31 14:22:56.987903 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:56.987874    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pztwr_bdce0f75-852b-49b6-9da3-f662bb82bf94/kube-rbac-proxy/0.log"
Oct 31 14:22:57.018450 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:57.018424    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gpl2l_7231ca35-6551-41ea-b341-46f8836425c1/dns-node-resolver/0.log"
Oct 31 14:22:57.796093 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:57.796056    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lgcv8_41b93592-66ac-4e0e-acf9-11271af65e89/node-ca/0.log"
Oct 31 14:22:58.645522 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:58.645493    4660 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-8kg4j"
Oct 31 14:22:58.980876 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:22:58.980794    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf_35162b05-708e-4e60-8fe0-7d78a8ca24ae/istio-proxy/0.log"
Oct 31 14:23:02.410951 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:02.410914    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-59fjz_97903974-f732-4729-9d22-d4d319663078/discovery/0.log"
Oct 31 14:23:02.515422 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:02.515393    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-2fpt9_ab9e59d3-2ac8-4381-98fa-22d3f25d682b/istio-proxy/0.log"
Oct 31 14:23:02.517613 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:02.517588    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-2fpt9_ab9e59d3-2ac8-4381-98fa-22d3f25d682b/istio-proxy/1.log"
Oct 31 14:23:02.521802 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:02.521751    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6bbdd9f58fn2phf_35162b05-708e-4e60-8fe0-7d78a8ca24ae/istio-proxy/0.log"
Oct 31 14:23:02.548691 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:02.548653    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-5z724_541446ef-8e36-4be7-9a16-b4005f6c250a/istio-proxy/0.log"
Oct 31 14:23:02.691752 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:02.691670    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-gwl2w_876ecf6e-3c48-4f96-8eaa-d7d5196be2ca/istio-proxy/0.log"
Oct 31 14:23:02.870084 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:02.870046    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6_26126910-4393-4b5e-b788-71db446afb82/istio-proxy/0.log"
Oct 31 14:23:02.982276 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:02.982143    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-848dcf9575-7trg4_c95a45a3-7757-4843-9161-6cdbf64385cd/router/0.log"
Oct 31 14:23:05.660752 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:05.660713    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l42ds_bd718975-d0cb-4a21-8383-c29164ae4f31/serve-healthcheck-canary/0.log"
Oct 31 14:23:05.977997 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:05.977915    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-59fjz_97903974-f732-4729-9d22-d4d319663078/discovery/0.log"
Oct 31 14:23:06.032258 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:06.032224    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-2fpt9_ab9e59d3-2ac8-4381-98fa-22d3f25d682b/istio-proxy/0.log"
Oct 31 14:23:06.035097 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:06.035062    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-2fpt9_ab9e59d3-2ac8-4381-98fa-22d3f25d682b/istio-proxy/1.log"
Oct 31 14:23:06.054361 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:06.054338    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-5z724_541446ef-8e36-4be7-9a16-b4005f6c250a/istio-proxy/0.log"
Oct 31 14:23:06.135574 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:06.135524    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-gwl2w_876ecf6e-3c48-4f96-8eaa-d7d5196be2ca/istio-proxy/0.log"
Oct 31 14:23:06.150829 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:06.150794    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-cpk9z_2a1a0fc2-36f2-4635-97b6-97faed580133/insights-operator/0.log"
Oct 31 14:23:06.153251 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:06.153224    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-cpk9z_2a1a0fc2-36f2-4635-97b6-97faed580133/insights-operator/1.log"
Oct 31 14:23:06.177269 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:06.177241    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9hjjp_aab0721f-f591-4a46-b56e-27848f31642f/kube-rbac-proxy/0.log"
Oct 31 14:23:06.203324 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:06.203281    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9hjjp_aab0721f-f591-4a46-b56e-27848f31642f/exporter/0.log"
Oct 31 14:23:06.224551 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:06.224525    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-54b87d5f7f-qt2c6_26126910-4393-4b5e-b788-71db446afb82/istio-proxy/0.log"
Oct 31 14:23:06.231086 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:06.231013    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9hjjp_aab0721f-f591-4a46-b56e-27848f31642f/extractor/0.log"
Oct 31 14:23:06.286552 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:06.286517    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-848dcf9575-7trg4_c95a45a3-7757-4843-9161-6cdbf64385cd/router/0.log"
Oct 31 14:23:06.465613 ip-10-0-137-77 kubenswrapper[4660]: I1031 14:23:06.465579    4660 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-7sff7-qhcml_45ff8b15-629b-481a-a3f1-595113718278/insights-gathering/0.log"